[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 08:02:53.590569 21084 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.151.62:39791,127.20.151.61:43053,127.20.151.60:34681
I20240501 08:02:53.591964 21084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:02:53.592886 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:02:53.609975 21093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:53.611871 21091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:53.612816 21084 server_base.cc:1034] running on GCE node
W20240501 08:02:53.610790 21090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:54.681768 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:02:54.681978 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:02:54.682121 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550574682105 us; error 0 us; skew 500 ppm
I20240501 08:02:54.682898 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:02:54.691908 21084 webserver.cc:460] Webserver started at http://127.20.151.62:36697/ using document root <none> and password file <none>
I20240501 08:02:54.692618 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:02:54.692796 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:02:54.693147 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:02:54.699020 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-0-root/instance:
uuid: "9f33e70baed74acdb0d488c5f9187ebc"
format_stamp: "Formatted at 2024-05-01 08:02:54 on dist-test-slave-pwkb"
I20240501 08:02:54.705871 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240501 08:02:54.710453 21099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:54.711397 21084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240501 08:02:54.711678 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-0-root
uuid: "9f33e70baed74acdb0d488c5f9187ebc"
format_stamp: "Formatted at 2024-05-01 08:02:54 on dist-test-slave-pwkb"
I20240501 08:02:54.711951 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:02:54.766140 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:02:54.767472 21084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:02:54.767894 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:02:54.833674 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:39791
I20240501 08:02:54.833808 21150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:39791 every 8 connection(s)
I20240501 08:02:54.839454 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:02:54.840327 21151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:02:54.846459 21153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:54.848896 21154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:54.850572 21156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:54.852932 21084 server_base.cc:1034] running on GCE node
I20240501 08:02:54.853574 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:02:54.853772 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:02:54.853927 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550574853911 us; error 0 us; skew 500 ppm
I20240501 08:02:54.854450 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:02:54.858628 21084 webserver.cc:460] Webserver started at http://127.20.151.61:34199/ using document root <none> and password file <none>
I20240501 08:02:54.859072 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:02:54.859241 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:02:54.859498 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:02:54.860471 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-1-root/instance:
uuid: "d12ee1ff2cdb4b0ab833eab4026defa5"
format_stamp: "Formatted at 2024-05-01 08:02:54 on dist-test-slave-pwkb"
I20240501 08:02:54.861624 21151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:02:54.865859 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240501 08:02:54.871417 21163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:54.872133 21084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:02:54.872385 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-1-root
uuid: "d12ee1ff2cdb4b0ab833eab4026defa5"
format_stamp: "Formatted at 2024-05-01 08:02:54 on dist-test-slave-pwkb"
I20240501 08:02:54.872635 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:02:54.887480 21151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:02:54.889544 21100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.61:43053: connect: Connection refused (error 111)
W20240501 08:02:54.893177 21151 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.151.61:43053: Network error: Client connection negotiation failed: client connection to 127.20.151.61:43053: connect: Connection refused (error 111)
I20240501 08:02:54.902966 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:02:54.904070 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:02:54.942670 21151 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } attempt: 1
W20240501 08:02:54.947796 21151 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.151.61:43053: Network error: Client connection negotiation failed: client connection to 127.20.151.61:43053: connect: Connection refused (error 111)
I20240501 08:02:54.967214 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.61:43053
I20240501 08:02:54.967411 21215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.61:43053 every 8 connection(s)
I20240501 08:02:54.971547 21216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:02:54.972395 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:02:54.977245 21216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:02:54.979265 21218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:54.981199 21219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:54.986029 21084 server_base.cc:1034] running on GCE node
W20240501 08:02:54.987294 21222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:54.988370 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:02:54.988629 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:02:54.988849 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550574988830 us; error 0 us; skew 500 ppm
I20240501 08:02:54.989461 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:02:54.991446 21216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:02:54.992614 21084 webserver.cc:460] Webserver started at http://127.20.151.60:41643/ using document root <none> and password file <none>
I20240501 08:02:54.993268 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:02:54.993474 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:02:54.993785 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:02:54.994932 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-2-root/instance:
uuid: "f618e172881f44c690fe61aab53e31a3"
format_stamp: "Formatted at 2024-05-01 08:02:54 on dist-test-slave-pwkb"
I20240501 08:02:55.000427 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20240501 08:02:55.005520 21228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:55.007802 21084 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20240501 08:02:55.008074 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-2-root
uuid: "f618e172881f44c690fe61aab53e31a3"
format_stamp: "Formatted at 2024-05-01 08:02:54 on dist-test-slave-pwkb"
I20240501 08:02:55.008457 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:02:55.010857 21216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:02:55.016314 21151 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } attempt: 2
W20240501 08:02:55.018460 21216 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.151.60:34681: Network error: Client connection negotiation failed: client connection to 127.20.151.60:34681: connect: Connection refused (error 111)
I20240501 08:02:55.024510 21151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:02:55.027842 21151 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.151.60:34681: Network error: Client connection negotiation failed: client connection to 127.20.151.60:34681: connect: Connection refused (error 111)
I20240501 08:02:55.031962 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:02:55.033109 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:02:55.059201 21151 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } attempt: 1
I20240501 08:02:55.061888 21216 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } attempt: 1
W20240501 08:02:55.063740 21151 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.151.60:34681: Network error: Client connection negotiation failed: client connection to 127.20.151.60:34681: connect: Connection refused (error 111)
W20240501 08:02:55.066815 21216 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.151.60:34681: Network error: Client connection negotiation failed: client connection to 127.20.151.60:34681: connect: Connection refused (error 111)
I20240501 08:02:55.097806 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.60:34681
I20240501 08:02:55.097908 21280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.60:34681 every 8 connection(s)
I20240501 08:02:55.102003 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:02:55.103026 21281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:02:55.107970 21281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:02:55.118880 21281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:02:55.127480 21281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:02:55.134269 21216 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } attempt: 2
I20240501 08:02:55.139199 21151 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } attempt: 2
I20240501 08:02:55.146309 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3: Bootstrap starting.
I20240501 08:02:55.153739 21281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3: Neither blocks nor log segments found. Creating new log.
I20240501 08:02:55.154641 21216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5: Bootstrap starting.
I20240501 08:02:55.154732 21151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc: Bootstrap starting.
I20240501 08:02:55.156064 21281 log.cc:824] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3: Log is configured to *not* fsync() on all Append() calls
I20240501 08:02:55.159826 21151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc: Neither blocks nor log segments found. Creating new log.
I20240501 08:02:55.160264 21216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5: Neither blocks nor log segments found. Creating new log.
I20240501 08:02:55.161978 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3: No bootstrap required, opened a new log
I20240501 08:02:55.165800 21216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5: No bootstrap required, opened a new log
I20240501 08:02:55.166993 21151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc: No bootstrap required, opened a new log
I20240501 08:02:55.183830 21216 raft_consensus.cc:348] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } }
I20240501 08:02:55.183843 21151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } }
I20240501 08:02:55.183964 21281 raft_consensus.cc:348] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } }
I20240501 08:02:55.184542 21216 raft_consensus.cc:374] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:02:55.184686 21151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:02:55.184795 21281 raft_consensus.cc:374] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:02:55.184855 21216 raft_consensus.cc:729] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12ee1ff2cdb4b0ab833eab4026defa5, State: Initialized, Role: FOLLOWER
I20240501 08:02:55.185001 21151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f33e70baed74acdb0d488c5f9187ebc, State: Initialized, Role: FOLLOWER
I20240501 08:02:55.185156 21281 raft_consensus.cc:729] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f618e172881f44c690fe61aab53e31a3, State: Initialized, Role: FOLLOWER
I20240501 08:02:55.185952 21151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } }
I20240501 08:02:55.186012 21281 consensus_queue.cc:260] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } }
I20240501 08:02:55.185950 21216 consensus_queue.cc:260] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } }
I20240501 08:02:55.190455 21289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } } }
I20240501 08:02:55.191434 21289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:02:55.191058 21291 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } } }
I20240501 08:02:55.192130 21291 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:02:55.192513 21151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:02:55.193379 21290 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } } }
I20240501 08:02:55.194234 21290 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:02:55.194259 21281 sys_catalog.cc:564] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:02:55.194797 21216 sys_catalog.cc:564] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:02:55.229019 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240501 08:02:55.239542 21318 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:02:55.239903 21318 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 08:02:55.239868 21321 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:02:55.240378 21321 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 08:02:55.242193 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240501 08:02:55.243613 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:02:55.247053 21325 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:02:55.247352 21325 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 08:02:55.252169 21326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:55.254354 21327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:55.257268 21329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:55.258800 21084 server_base.cc:1034] running on GCE node
I20240501 08:02:55.259479 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:02:55.259637 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:02:55.259810 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550575259790 us; error 0 us; skew 500 ppm
I20240501 08:02:55.260293 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:02:55.262689 21084 webserver.cc:460] Webserver started at http://127.20.151.1:44625/ using document root <none> and password file <none>
I20240501 08:02:55.263139 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:02:55.263314 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:02:55.263571 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:02:55.264590 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-0-root/instance:
uuid: "36549675d0eb443f8858f141f67dc941"
format_stamp: "Formatted at 2024-05-01 08:02:55 on dist-test-slave-pwkb"
I20240501 08:02:55.269040 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:02:55.272346 21334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20240501 08:02:55.272801 21105 tablet.cc:2369] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:02:55.273236 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:02:55.273505 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-0-root
uuid: "36549675d0eb443f8858f141f67dc941"
format_stamp: "Formatted at 2024-05-01 08:02:55 on dist-test-slave-pwkb"
I20240501 08:02:55.273841 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:02:55.311941 21291 raft_consensus.cc:482] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:02:55.312638 21291 raft_consensus.cc:504] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } }
I20240501 08:02:55.315699 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:02:55.316428 21291 leader_election.cc:288] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f33e70baed74acdb0d488c5f9187ebc (127.20.151.62:39791), d12ee1ff2cdb4b0ab833eab4026defa5 (127.20.151.61:43053)
I20240501 08:02:55.316484 21191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f618e172881f44c690fe61aab53e31a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" is_pre_election: true
I20240501 08:02:55.317322 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:02:55.317399 21191 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f618e172881f44c690fe61aab53e31a3 in term 0.
I20240501 08:02:55.318589 21126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f618e172881f44c690fe61aab53e31a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f33e70baed74acdb0d488c5f9187ebc" is_pre_election: true
I20240501 08:02:55.318984 21229 leader_election.cc:302] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d12ee1ff2cdb4b0ab833eab4026defa5, f618e172881f44c690fe61aab53e31a3; no voters:
I20240501 08:02:55.319327 21126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f618e172881f44c690fe61aab53e31a3 in term 0.
I20240501 08:02:55.320147 21291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:02:55.320597 21291 raft_consensus.cc:482] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:02:55.321024 21291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:02:55.321519 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:02:55.326632 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:02:55.326897 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:55.327190 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:02:55.327394 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:55.327829 21291 raft_consensus.cc:504] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } }
I20240501 08:02:55.330686 21126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f618e172881f44c690fe61aab53e31a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f33e70baed74acdb0d488c5f9187ebc"
I20240501 08:02:55.330935 21191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f618e172881f44c690fe61aab53e31a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5"
I20240501 08:02:55.331240 21126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:02:55.331535 21191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:02:55.337970 21126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f618e172881f44c690fe61aab53e31a3 in term 1.
I20240501 08:02:55.338301 21191 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f618e172881f44c690fe61aab53e31a3 in term 1.
I20240501 08:02:55.339326 21233 leader_election.cc:302] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f33e70baed74acdb0d488c5f9187ebc, f618e172881f44c690fe61aab53e31a3; no voters:
I20240501 08:02:55.339932 21291 leader_election.cc:288] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [CANDIDATE]: Term 1 election: Requested vote from peers 9f33e70baed74acdb0d488c5f9187ebc (127.20.151.62:39791), d12ee1ff2cdb4b0ab833eab4026defa5 (127.20.151.61:43053)
I20240501 08:02:55.340587 21291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:02:55.341130 21291 raft_consensus.cc:686] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 1 LEADER]: Becoming Leader. State: Replica: f618e172881f44c690fe61aab53e31a3, State: Running, Role: LEADER
I20240501 08:02:55.342698 21291 consensus_queue.cc:237] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } }
I20240501 08:02:55.356034 21291 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f618e172881f44c690fe61aab53e31a3. Latest consensus state: current_term: 1 leader_uuid: "f618e172881f44c690fe61aab53e31a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } } }
I20240501 08:02:55.357069 21291 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [sys.catalog]: This master's current role is: LEADER
I20240501 08:02:55.365228 21347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:02:55.375097 21347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:02:55.396461 21191 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [term 1 FOLLOWER]: Refusing update from remote peer f618e172881f44c690fe61aab53e31a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:02:55.397583 21126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [term 1 FOLLOWER]: Refusing update from remote peer f618e172881f44c690fe61aab53e31a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:02:55.398453 21291 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:02:55.399863 21343 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:02:55.426406 21289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f618e172881f44c690fe61aab53e31a3. Latest consensus state: current_term: 1 leader_uuid: "f618e172881f44c690fe61aab53e31a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } } }
I20240501 08:02:55.427430 21289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:02:55.436179 21361 mvcc.cc:205] Tried to move back new op lower bound from 7022799156800385024 to 7022799156620922880. Current Snapshot: MvccSnapshot[applied={T|T < 7022799156800385024}]
I20240501 08:02:55.439392 21290 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f618e172881f44c690fe61aab53e31a3. Latest consensus state: current_term: 1 leader_uuid: "f618e172881f44c690fe61aab53e31a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } } }
I20240501 08:02:55.440168 21290 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:02:55.452920 21289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f618e172881f44c690fe61aab53e31a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } } }
I20240501 08:02:55.454375 21289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:02:55.455446 21290 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f618e172881f44c690fe61aab53e31a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } } }
I20240501 08:02:55.456398 21290 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:02:55.460429 21343 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f618e172881f44c690fe61aab53e31a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } } }
I20240501 08:02:55.461407 21343 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [sys.catalog]: This master's current role is: LEADER
I20240501 08:02:55.462188 21343 sys_catalog.cc:455] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f618e172881f44c690fe61aab53e31a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f33e70baed74acdb0d488c5f9187ebc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39791 } } peers { permanent_uuid: "d12ee1ff2cdb4b0ab833eab4026defa5" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 43053 } } peers { permanent_uuid: "f618e172881f44c690fe61aab53e31a3" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34681 } } }
I20240501 08:02:55.463065 21343 sys_catalog.cc:458] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [sys.catalog]: This master's current role is: LEADER
I20240501 08:02:55.465711 21347 catalog_manager.cc:1347] Generated new cluster ID: cd4e759f39aa4779879345b761e8ffb6
I20240501 08:02:55.466089 21347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:02:55.539006 21347 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:02:55.541160 21347 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:02:55.567111 21347 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3: Generated new TSK 0
I20240501 08:02:55.568080 21347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:02:55.589363 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:37949
I20240501 08:02:55.589601 21408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:37949 every 8 connection(s)
I20240501 08:02:55.614924 21409 heartbeater.cc:347] Connected to a master server at 127.20.151.60:34681
I20240501 08:02:55.615511 21409 heartbeater.cc:464] Registering TS with master...
I20240501 08:02:55.616921 21409 heartbeater.cc:511] Master 127.20.151.60:34681 requested a full tablet report, sending...
I20240501 08:02:55.620954 21246 ts_manager.cc:194] Registered new tserver with Master: 36549675d0eb443f8858f141f67dc941 (127.20.151.1:37949)
I20240501 08:02:55.621556 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:02:55.627148 21410 heartbeater.cc:347] Connected to a master server at 127.20.151.61:43053
I20240501 08:02:55.627638 21410 heartbeater.cc:464] Registering TS with master...
I20240501 08:02:55.628865 21410 heartbeater.cc:511] Master 127.20.151.61:43053 requested a full tablet report, sending...
I20240501 08:02:55.624418 21246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52322
I20240501 08:02:55.634086 21181 ts_manager.cc:194] Registered new tserver with Master: 36549675d0eb443f8858f141f67dc941 (127.20.151.1:37949)
I20240501 08:02:55.642134 21412 heartbeater.cc:347] Connected to a master server at 127.20.151.62:39791
I20240501 08:02:55.642591 21412 heartbeater.cc:464] Registering TS with master...
I20240501 08:02:55.643369 21412 heartbeater.cc:511] Master 127.20.151.62:39791 requested a full tablet report, sending...
W20240501 08:02:55.644973 21416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:55.645828 21417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:55.646615 21116 ts_manager.cc:194] Registered new tserver with Master: 36549675d0eb443f8858f141f67dc941 (127.20.151.1:37949)
W20240501 08:02:55.650655 21419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:55.656046 21084 server_base.cc:1034] running on GCE node
I20240501 08:02:55.656834 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:02:55.657001 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:02:55.657158 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550575657143 us; error 0 us; skew 500 ppm
I20240501 08:02:55.657620 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:02:55.660359 21084 webserver.cc:460] Webserver started at http://127.20.151.2:45165/ using document root <none> and password file <none>
I20240501 08:02:55.660748 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:02:55.660921 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:02:55.661156 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:02:55.662153 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-1-root/instance:
uuid: "a370e7822fd146af95528ef2d5d4be95"
format_stamp: "Formatted at 2024-05-01 08:02:55 on dist-test-slave-pwkb"
I20240501 08:02:55.666437 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 08:02:55.669333 21424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:55.670078 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240501 08:02:55.670321 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-1-root
uuid: "a370e7822fd146af95528ef2d5d4be95"
format_stamp: "Formatted at 2024-05-01 08:02:55 on dist-test-slave-pwkb"
I20240501 08:02:55.670565 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:02:55.688385 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:02:55.689407 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:02:55.690899 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:02:55.693462 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:02:55.693650 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:55.693913 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:02:55.694046 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:55.782610 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:34535
I20240501 08:02:55.782739 21486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:34535 every 8 connection(s)
I20240501 08:02:55.808614 21487 heartbeater.cc:347] Connected to a master server at 127.20.151.60:34681
I20240501 08:02:55.809294 21487 heartbeater.cc:464] Registering TS with master...
I20240501 08:02:55.809641 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:02:55.810767 21487 heartbeater.cc:511] Master 127.20.151.60:34681 requested a full tablet report, sending...
I20240501 08:02:55.813062 21246 ts_manager.cc:194] Registered new tserver with Master: a370e7822fd146af95528ef2d5d4be95 (127.20.151.2:34535)
I20240501 08:02:55.822926 21246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52332
I20240501 08:02:55.823429 21490 heartbeater.cc:347] Connected to a master server at 127.20.151.62:39791
I20240501 08:02:55.823830 21490 heartbeater.cc:464] Registering TS with master...
I20240501 08:02:55.824707 21490 heartbeater.cc:511] Master 127.20.151.62:39791 requested a full tablet report, sending...
I20240501 08:02:55.826357 21488 heartbeater.cc:347] Connected to a master server at 127.20.151.61:43053
I20240501 08:02:55.826756 21488 heartbeater.cc:464] Registering TS with master...
I20240501 08:02:55.827466 21488 heartbeater.cc:511] Master 127.20.151.61:43053 requested a full tablet report, sending...
W20240501 08:02:55.828558 21494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:55.830008 21116 ts_manager.cc:194] Registered new tserver with Master: a370e7822fd146af95528ef2d5d4be95 (127.20.151.2:34535)
I20240501 08:02:55.833660 21181 ts_manager.cc:194] Registered new tserver with Master: a370e7822fd146af95528ef2d5d4be95 (127.20.151.2:34535)
W20240501 08:02:55.836316 21495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:55.839095 21497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:55.840940 21084 server_base.cc:1034] running on GCE node
I20240501 08:02:55.841895 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:02:55.842093 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:02:55.842255 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550575842238 us; error 0 us; skew 500 ppm
I20240501 08:02:55.842713 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:02:55.845492 21084 webserver.cc:460] Webserver started at http://127.20.151.3:34581/ using document root <none> and password file <none>
I20240501 08:02:55.845973 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:02:55.846156 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:02:55.846407 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:02:55.847455 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-2-root/instance:
uuid: "f1dc0b5c7c9741efa002e131bf576b4e"
format_stamp: "Formatted at 2024-05-01 08:02:55 on dist-test-slave-pwkb"
I20240501 08:02:55.851807 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240501 08:02:55.855028 21502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:55.855762 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:02:55.856004 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-2-root
uuid: "f1dc0b5c7c9741efa002e131bf576b4e"
format_stamp: "Formatted at 2024-05-01 08:02:55 on dist-test-slave-pwkb"
I20240501 08:02:55.856235 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:02:55.867473 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:02:55.868582 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:02:55.870952 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:02:55.874445 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:02:55.874774 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:55.875021 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:02:55.875231 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:55.968987 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:39231
I20240501 08:02:55.969105 21564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:39231 every 8 connection(s)
I20240501 08:02:55.998713 21565 heartbeater.cc:347] Connected to a master server at 127.20.151.60:34681
I20240501 08:02:55.999050 21565 heartbeater.cc:464] Registering TS with master...
I20240501 08:02:55.999962 21565 heartbeater.cc:511] Master 127.20.151.60:34681 requested a full tablet report, sending...
I20240501 08:02:56.002005 21246 ts_manager.cc:194] Registered new tserver with Master: f1dc0b5c7c9741efa002e131bf576b4e (127.20.151.3:39231)
I20240501 08:02:56.003691 21246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52348
I20240501 08:02:56.005146 21567 heartbeater.cc:347] Connected to a master server at 127.20.151.62:39791
I20240501 08:02:56.005484 21567 heartbeater.cc:464] Registering TS with master...
I20240501 08:02:56.006170 21567 heartbeater.cc:511] Master 127.20.151.62:39791 requested a full tablet report, sending...
I20240501 08:02:56.008387 21116 ts_manager.cc:194] Registered new tserver with Master: f1dc0b5c7c9741efa002e131bf576b4e (127.20.151.3:39231)
I20240501 08:02:56.013021 21566 heartbeater.cc:347] Connected to a master server at 127.20.151.61:43053
I20240501 08:02:56.013398 21566 heartbeater.cc:464] Registering TS with master...
I20240501 08:02:56.014103 21566 heartbeater.cc:511] Master 127.20.151.61:43053 requested a full tablet report, sending...
I20240501 08:02:56.016165 21181 ts_manager.cc:194] Registered new tserver with Master: f1dc0b5c7c9741efa002e131bf576b4e (127.20.151.3:39231)
I20240501 08:02:56.017204 21084 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.023139958s
I20240501 08:02:56.244170 21318 catalog_manager.cc:1259] Loaded cluster ID: cd4e759f39aa4779879345b761e8ffb6
I20240501 08:02:56.244414 21318 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc: loading cluster ID for follower catalog manager: success
I20240501 08:02:56.248967 21318 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc: acquiring CA information for follower catalog manager: success
I20240501 08:02:56.252533 21325 catalog_manager.cc:1259] Loaded cluster ID: cd4e759f39aa4779879345b761e8ffb6
I20240501 08:02:56.252568 21318 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 08:02:56.253008 21325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5: loading cluster ID for follower catalog manager: success
I20240501 08:02:56.259403 21325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5: acquiring CA information for follower catalog manager: success
I20240501 08:02:56.264096 21325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 08:02:56.640059 21409 heartbeater.cc:503] Master 127.20.151.60:34681 was elected leader, sending a full tablet report...
I20240501 08:02:56.825786 21487 heartbeater.cc:503] Master 127.20.151.60:34681 was elected leader, sending a full tablet report...
I20240501 08:02:57.006287 21565 heartbeater.cc:503] Master 127.20.151.60:34681 was elected leader, sending a full tablet report...
I20240501 08:02:57.231618 21319 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:02:57.325431 21084 test_util.cc:268] Using random seed: -367237795
I20240501 08:02:57.388038 21246 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52354:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 08:02:57.476228 21530 tablet_service.cc:1467] Processing CreateTablet for tablet 76b21384be2d4d71a1a3de4c6980527f (DEFAULT_TABLE table=test-workload [id=5aa425f1b0574b98b1ab67d404f3213c]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 08:02:57.477766 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b21384be2d4d71a1a3de4c6980527f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:02:57.478796 21366 tablet_service.cc:1467] Processing CreateTablet for tablet 402d9463b9ac461a940155a46cb5df09 (DEFAULT_TABLE table=test-workload [id=5aa425f1b0574b98b1ab67d404f3213c]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 08:02:57.479151 21365 tablet_service.cc:1467] Processing CreateTablet for tablet 9df94a39c5184def8a92ffcd233930e8 (DEFAULT_TABLE table=test-workload [id=5aa425f1b0574b98b1ab67d404f3213c]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 08:02:57.480660 21366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 402d9463b9ac461a940155a46cb5df09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:02:57.480672 21452 tablet_service.cc:1467] Processing CreateTablet for tablet b378e37e5f27412babb25df9d2892593 (DEFAULT_TABLE table=test-workload [id=5aa425f1b0574b98b1ab67d404f3213c]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 08:02:57.481357 21365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df94a39c5184def8a92ffcd233930e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:02:57.482196 21452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b378e37e5f27412babb25df9d2892593. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:02:57.499735 21601 tablet_bootstrap.cc:492] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e: Bootstrap starting.
I20240501 08:02:57.500969 21603 tablet_bootstrap.cc:492] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95: Bootstrap starting.
I20240501 08:02:57.501590 21602 tablet_bootstrap.cc:492] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941: Bootstrap starting.
I20240501 08:02:57.506477 21603 tablet_bootstrap.cc:654] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95: Neither blocks nor log segments found. Creating new log.
I20240501 08:02:57.508858 21602 tablet_bootstrap.cc:654] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941: Neither blocks nor log segments found. Creating new log.
I20240501 08:02:57.508879 21601 tablet_bootstrap.cc:654] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e: Neither blocks nor log segments found. Creating new log.
I20240501 08:02:57.512228 21603 tablet_bootstrap.cc:492] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95: No bootstrap required, opened a new log
I20240501 08:02:57.512687 21603 ts_tablet_manager.cc:1367] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240501 08:02:57.515699 21603 raft_consensus.cc:348] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a370e7822fd146af95528ef2d5d4be95" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34535 } }
I20240501 08:02:57.516366 21603 raft_consensus.cc:374] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:02:57.516549 21601 tablet_bootstrap.cc:492] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e: No bootstrap required, opened a new log
I20240501 08:02:57.517066 21601 ts_tablet_manager.cc:1367] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20240501 08:02:57.517635 21603 raft_consensus.cc:729] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a370e7822fd146af95528ef2d5d4be95, State: Initialized, Role: FOLLOWER
I20240501 08:02:57.518774 21602 tablet_bootstrap.cc:492] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941: No bootstrap required, opened a new log
I20240501 08:02:57.518558 21603 consensus_queue.cc:260] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a370e7822fd146af95528ef2d5d4be95" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34535 } }
I20240501 08:02:57.519143 21602 ts_tablet_manager.cc:1367] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20240501 08:02:57.519240 21603 raft_consensus.cc:388] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:02:57.519630 21603 raft_consensus.cc:482] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:02:57.519616 21601 raft_consensus.cc:348] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dc0b5c7c9741efa002e131bf576b4e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 39231 } }
I20240501 08:02:57.520128 21603 raft_consensus.cc:3037] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:02:57.520321 21601 raft_consensus.cc:374] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:02:57.520663 21601 raft_consensus.cc:729] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1dc0b5c7c9741efa002e131bf576b4e, State: Initialized, Role: FOLLOWER
I20240501 08:02:57.521065 21602 raft_consensus.cc:348] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36549675d0eb443f8858f141f67dc941" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37949 } }
I20240501 08:02:57.521549 21602 raft_consensus.cc:374] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:02:57.521317 21601 consensus_queue.cc:260] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dc0b5c7c9741efa002e131bf576b4e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 39231 } }
I20240501 08:02:57.521809 21602 raft_consensus.cc:729] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36549675d0eb443f8858f141f67dc941, State: Initialized, Role: FOLLOWER
I20240501 08:02:57.522053 21601 raft_consensus.cc:388] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:02:57.522454 21601 raft_consensus.cc:482] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:02:57.522881 21601 raft_consensus.cc:3037] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:02:57.522397 21602 consensus_queue.cc:260] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36549675d0eb443f8858f141f67dc941" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37949 } }
I20240501 08:02:57.525907 21602 raft_consensus.cc:388] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:02:57.526225 21602 raft_consensus.cc:482] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:02:57.526512 21602 raft_consensus.cc:3037] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:02:57.526616 21603 raft_consensus.cc:504] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a370e7822fd146af95528ef2d5d4be95" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34535 } }
I20240501 08:02:57.527384 21603 leader_election.cc:302] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a370e7822fd146af95528ef2d5d4be95; no voters:
I20240501 08:02:57.529336 21601 raft_consensus.cc:504] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dc0b5c7c9741efa002e131bf576b4e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 39231 } }
I20240501 08:02:57.530609 21603 leader_election.cc:288] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:02:57.531057 21607 raft_consensus.cc:2781] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:02:57.531142 21601 leader_election.cc:302] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1dc0b5c7c9741efa002e131bf576b4e; no voters:
I20240501 08:02:57.532244 21602 raft_consensus.cc:504] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36549675d0eb443f8858f141f67dc941" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37949 } }
I20240501 08:02:57.533129 21602 leader_election.cc:302] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36549675d0eb443f8858f141f67dc941; no voters:
I20240501 08:02:57.537093 21610 raft_consensus.cc:2781] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:02:57.537619 21601 leader_election.cc:288] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:02:57.539029 21607 raft_consensus.cc:686] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [term 1 LEADER]: Becoming Leader. State: Replica: a370e7822fd146af95528ef2d5d4be95, State: Running, Role: LEADER
I20240501 08:02:57.540088 21607 consensus_queue.cc:237] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a370e7822fd146af95528ef2d5d4be95" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34535 } }
I20240501 08:02:57.543694 21603 ts_tablet_manager.cc:1398] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95: Time spent starting tablet: real 0.031s user 0.014s sys 0.014s
I20240501 08:02:57.549108 21602 leader_election.cc:288] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:02:57.551882 21610 raft_consensus.cc:686] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [term 1 LEADER]: Becoming Leader. State: Replica: f1dc0b5c7c9741efa002e131bf576b4e, State: Running, Role: LEADER
I20240501 08:02:57.552913 21610 consensus_queue.cc:237] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dc0b5c7c9741efa002e131bf576b4e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 39231 } }
I20240501 08:02:57.557909 21601 ts_tablet_manager.cc:1398] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e: Time spent starting tablet: real 0.040s user 0.017s sys 0.008s
I20240501 08:02:57.566593 21246 catalog_manager.cc:5484] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 reported cstate change: term changed from 0 to 1, leader changed from <none> to a370e7822fd146af95528ef2d5d4be95 (127.20.151.2). New cstate: current_term: 1 leader_uuid: "a370e7822fd146af95528ef2d5d4be95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a370e7822fd146af95528ef2d5d4be95" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34535 } health_report { overall_health: HEALTHY } } }
I20240501 08:02:57.580713 21245 catalog_manager.cc:5484] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e reported cstate change: term changed from 0 to 1, leader changed from <none> to f1dc0b5c7c9741efa002e131bf576b4e (127.20.151.3). New cstate: current_term: 1 leader_uuid: "f1dc0b5c7c9741efa002e131bf576b4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dc0b5c7c9741efa002e131bf576b4e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 39231 } health_report { overall_health: HEALTHY } } }
I20240501 08:02:57.612038 21611 raft_consensus.cc:2781] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:02:57.615976 21611 raft_consensus.cc:686] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [term 1 LEADER]: Becoming Leader. State: Replica: 36549675d0eb443f8858f141f67dc941, State: Running, Role: LEADER
I20240501 08:02:57.616783 21611 consensus_queue.cc:237] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36549675d0eb443f8858f141f67dc941" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37949 } }
I20240501 08:02:57.631389 21602 ts_tablet_manager.cc:1398] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941: Time spent starting tablet: real 0.112s user 0.046s sys 0.061s
I20240501 08:02:57.632556 21602 tablet_bootstrap.cc:492] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941: Bootstrap starting.
I20240501 08:02:57.638527 21602 tablet_bootstrap.cc:654] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941: Neither blocks nor log segments found. Creating new log.
I20240501 08:02:57.639393 21245 catalog_manager.cc:5484] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36549675d0eb443f8858f141f67dc941 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "36549675d0eb443f8858f141f67dc941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36549675d0eb443f8858f141f67dc941" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37949 } health_report { overall_health: HEALTHY } } }
I20240501 08:02:57.644999 21602 tablet_bootstrap.cc:492] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941: No bootstrap required, opened a new log
I20240501 08:02:57.645491 21602 ts_tablet_manager.cc:1367] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20240501 08:02:57.648106 21602 raft_consensus.cc:348] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36549675d0eb443f8858f141f67dc941" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37949 } }
I20240501 08:02:57.648681 21602 raft_consensus.cc:374] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:02:57.648983 21602 raft_consensus.cc:729] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36549675d0eb443f8858f141f67dc941, State: Initialized, Role: FOLLOWER
I20240501 08:02:57.649614 21602 consensus_queue.cc:260] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36549675d0eb443f8858f141f67dc941" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37949 } }
I20240501 08:02:57.650295 21602 raft_consensus.cc:388] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:02:57.650628 21602 raft_consensus.cc:482] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:02:57.650985 21602 raft_consensus.cc:3037] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:02:57.656994 21602 raft_consensus.cc:504] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36549675d0eb443f8858f141f67dc941" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37949 } }
I20240501 08:02:57.657625 21602 leader_election.cc:302] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36549675d0eb443f8858f141f67dc941; no voters:
I20240501 08:02:57.658131 21602 leader_election.cc:288] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:02:57.658288 21611 raft_consensus.cc:2781] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:02:57.659060 21611 raft_consensus.cc:686] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [term 1 LEADER]: Becoming Leader. State: Replica: 36549675d0eb443f8858f141f67dc941, State: Running, Role: LEADER
I20240501 08:02:57.659503 21602 ts_tablet_manager.cc:1398] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20240501 08:02:57.659749 21611 consensus_queue.cc:237] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36549675d0eb443f8858f141f67dc941" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37949 } }
I20240501 08:02:57.676707 21245 catalog_manager.cc:5484] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36549675d0eb443f8858f141f67dc941 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "36549675d0eb443f8858f141f67dc941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36549675d0eb443f8858f141f67dc941" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37949 } health_report { overall_health: HEALTHY } } }
I20240501 08:02:57.717949 21084 tablet_server.cc:178] TabletServer@127.20.151.1:0 shutting down...
I20240501 08:02:57.737018 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:02:57.737850 21084 tablet_replica.cc:330] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941: stopping tablet replica
I20240501 08:02:57.738415 21084 raft_consensus.cc:2229] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:02:57.738845 21084 raft_consensus.cc:2258] T 9df94a39c5184def8a92ffcd233930e8 P 36549675d0eb443f8858f141f67dc941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:02:57.741273 21084 tablet_replica.cc:330] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941: stopping tablet replica
I20240501 08:02:57.741771 21084 raft_consensus.cc:2229] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:02:57.742139 21084 raft_consensus.cc:2258] T 402d9463b9ac461a940155a46cb5df09 P 36549675d0eb443f8858f141f67dc941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:02:57.884754 21084 tablet_server.cc:194] TabletServer@127.20.151.1:0 shutdown complete.
I20240501 08:02:57.893296 21084 tablet_server.cc:178] TabletServer@127.20.151.2:0 shutting down...
I20240501 08:02:57.914907 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:02:57.915637 21084 tablet_replica.cc:330] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95: stopping tablet replica
I20240501 08:02:57.916070 21084 raft_consensus.cc:2229] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:02:57.916446 21084 raft_consensus.cc:2258] T b378e37e5f27412babb25df9d2892593 P a370e7822fd146af95528ef2d5d4be95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:02:57.935191 21084 tablet_server.cc:194] TabletServer@127.20.151.2:0 shutdown complete.
I20240501 08:02:57.941725 21084 tablet_server.cc:178] TabletServer@127.20.151.3:0 shutting down...
I20240501 08:02:57.955370 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:02:57.955988 21084 tablet_replica.cc:330] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e: stopping tablet replica
I20240501 08:02:57.956779 21084 raft_consensus.cc:2229] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:02:57.957118 21084 raft_consensus.cc:2258] T 76b21384be2d4d71a1a3de4c6980527f P f1dc0b5c7c9741efa002e131bf576b4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:02:57.975759 21084 tablet_server.cc:194] TabletServer@127.20.151.3:0 shutdown complete.
I20240501 08:02:57.982528 21084 master.cc:537] Master@127.20.151.62:39791 shutting down...
I20240501 08:02:58.084820 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:02:58.085327 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:02:58.085701 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9f33e70baed74acdb0d488c5f9187ebc: stopping tablet replica
I20240501 08:02:58.103500 21084 master.cc:552] Master@127.20.151.62:39791 shutdown complete.
I20240501 08:02:58.141728 21084 master.cc:537] Master@127.20.151.61:43053 shutting down...
I20240501 08:02:58.161912 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:02:58.162449 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:02:58.162779 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P d12ee1ff2cdb4b0ab833eab4026defa5: stopping tablet replica
I20240501 08:02:58.209192 21084 master.cc:552] Master@127.20.151.61:43053 shutdown complete.
I20240501 08:02:58.220808 21084 master.cc:537] Master@127.20.151.60:34681 shutting down...
W20240501 08:02:58.236732 21233 consensus_peers.cc:485] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 -> Peer 9f33e70baed74acdb0d488c5f9187ebc (127.20.151.62:39791): Couldn't send request to peer 9f33e70baed74acdb0d488c5f9187ebc. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.62:39791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:02:58.237093 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:02:58.237905 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:02:58.238183 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P f618e172881f44c690fe61aab53e31a3: stopping tablet replica
I20240501 08:02:58.255074 21084 master.cc:552] Master@127.20.151.60:34681 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (4692 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240501 08:02:58.280109 21084 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.151.62:34829
I20240501 08:02:58.280963 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:02:58.285800 21626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:58.286820 21627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:58.287711 21629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:58.288270 21084 server_base.cc:1034] running on GCE node
I20240501 08:02:58.288911 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:02:58.289070 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:02:58.289176 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550578289166 us; error 0 us; skew 500 ppm
I20240501 08:02:58.289602 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:02:58.291878 21084 webserver.cc:460] Webserver started at http://127.20.151.62:34211/ using document root <none> and password file <none>
I20240501 08:02:58.292256 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:02:58.292392 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:02:58.292586 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:02:58.293550 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/master-0-root/instance:
uuid: "56ec874134e24b8fa93635893c66c655"
format_stamp: "Formatted at 2024-05-01 08:02:58 on dist-test-slave-pwkb"
I20240501 08:02:58.297363 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:02:58.300253 21634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:58.300896 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 08:02:58.301124 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/master-0-root
uuid: "56ec874134e24b8fa93635893c66c655"
format_stamp: "Formatted at 2024-05-01 08:02:58 on dist-test-slave-pwkb"
I20240501 08:02:58.301345 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:02:58.320905 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:02:58.321805 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:02:58.353489 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:34829
I20240501 08:02:58.353581 21685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:34829 every 8 connection(s)
I20240501 08:02:58.356858 21686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:02:58.365406 21686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655: Bootstrap starting.
I20240501 08:02:58.368811 21686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655: Neither blocks nor log segments found. Creating new log.
I20240501 08:02:58.372017 21686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655: No bootstrap required, opened a new log
I20240501 08:02:58.373601 21686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ec874134e24b8fa93635893c66c655" member_type: VOTER }
I20240501 08:02:58.374011 21686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:02:58.374214 21686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ec874134e24b8fa93635893c66c655, State: Initialized, Role: FOLLOWER
I20240501 08:02:58.374672 21686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ec874134e24b8fa93635893c66c655" member_type: VOTER }
I20240501 08:02:58.375087 21686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:02:58.375283 21686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:02:58.375495 21686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:02:58.378906 21686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ec874134e24b8fa93635893c66c655" member_type: VOTER }
I20240501 08:02:58.379344 21686 leader_election.cc:302] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56ec874134e24b8fa93635893c66c655; no voters:
I20240501 08:02:58.380282 21686 leader_election.cc:288] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:02:58.380580 21689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:02:58.381851 21689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [term 1 LEADER]: Becoming Leader. State: Replica: 56ec874134e24b8fa93635893c66c655, State: Running, Role: LEADER
I20240501 08:02:58.382467 21689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ec874134e24b8fa93635893c66c655" member_type: VOTER }
I20240501 08:02:58.382905 21686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:02:58.385008 21691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56ec874134e24b8fa93635893c66c655. Latest consensus state: current_term: 1 leader_uuid: "56ec874134e24b8fa93635893c66c655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ec874134e24b8fa93635893c66c655" member_type: VOTER } }
I20240501 08:02:58.384986 21690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56ec874134e24b8fa93635893c66c655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ec874134e24b8fa93635893c66c655" member_type: VOTER } }
I20240501 08:02:58.385638 21691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [sys.catalog]: This master's current role is: LEADER
I20240501 08:02:58.385789 21690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [sys.catalog]: This master's current role is: LEADER
I20240501 08:02:58.389436 21694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:02:58.395311 21694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:02:58.396261 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:02:58.403246 21694 catalog_manager.cc:1347] Generated new cluster ID: e26f5f46e8314be9a111687c77b87602
I20240501 08:02:58.403509 21694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:02:58.428089 21694 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:02:58.429186 21694 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:02:58.442174 21694 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655: Generated new TSK 0
I20240501 08:02:58.442682 21694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:02:58.462404 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:02:58.467862 21707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:58.469803 21708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:58.470961 21710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:58.474556 21084 server_base.cc:1034] running on GCE node
I20240501 08:02:58.475883 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:02:58.476049 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:02:58.476195 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550578476173 us; error 0 us; skew 500 ppm
I20240501 08:02:58.476632 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:02:58.478709 21084 webserver.cc:460] Webserver started at http://127.20.151.1:37479/ using document root <none> and password file <none>
I20240501 08:02:58.479071 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:02:58.479243 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:02:58.479463 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:02:58.480404 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-0-root/instance:
uuid: "01d3577427624fa8b733111b0dc0d119"
format_stamp: "Formatted at 2024-05-01 08:02:58 on dist-test-slave-pwkb"
I20240501 08:02:58.484619 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:02:58.487462 21715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:58.488127 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 08:02:58.488588 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-0-root
uuid: "01d3577427624fa8b733111b0dc0d119"
format_stamp: "Formatted at 2024-05-01 08:02:58 on dist-test-slave-pwkb"
I20240501 08:02:58.488852 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:02:58.506567 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:02:58.507369 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:02:58.508634 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:02:58.510753 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:02:58.510942 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:58.511150 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:02:58.511281 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:58.544682 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:37279
I20240501 08:02:58.544770 21777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:37279 every 8 connection(s)
I20240501 08:02:58.548692 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:02:58.555794 21782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:58.557161 21783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:58.559454 21785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:58.560508 21084 server_base.cc:1034] running on GCE node
I20240501 08:02:58.561081 21778 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34829
I20240501 08:02:58.561342 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 08:02:58.561380 21778 heartbeater.cc:464] Registering TS with master...
W20240501 08:02:58.561674 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:02:58.561897 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550578561878 us; error 0 us; skew 500 ppm
I20240501 08:02:58.562299 21778 heartbeater.cc:511] Master 127.20.151.62:34829 requested a full tablet report, sending...
I20240501 08:02:58.562431 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:02:58.564384 21651 ts_manager.cc:194] Registered new tserver with Master: 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279)
I20240501 08:02:58.565227 21084 webserver.cc:460] Webserver started at http://127.20.151.2:41455/ using document root <none> and password file <none>
I20240501 08:02:58.565863 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:02:58.566097 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:02:58.566395 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:02:58.566749 21651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59120
I20240501 08:02:58.567715 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-1-root/instance:
uuid: "8ab12efc498b426697bb1a5531064c2f"
format_stamp: "Formatted at 2024-05-01 08:02:58 on dist-test-slave-pwkb"
I20240501 08:02:58.571674 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20240501 08:02:58.574537 21790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:58.575312 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:02:58.575536 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-1-root
uuid: "8ab12efc498b426697bb1a5531064c2f"
format_stamp: "Formatted at 2024-05-01 08:02:58 on dist-test-slave-pwkb"
I20240501 08:02:58.575747 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:02:58.586481 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:02:58.587292 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:02:58.588522 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:02:58.590723 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:02:58.590929 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:58.591153 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:02:58.591303 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:58.625488 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:44959
I20240501 08:02:58.625576 21852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:44959 every 8 connection(s)
I20240501 08:02:58.629789 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:02:58.636159 21856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:58.637754 21857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:02:58.640730 21859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:02:58.641769 21853 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34829
I20240501 08:02:58.642048 21084 server_base.cc:1034] running on GCE node
I20240501 08:02:58.642061 21853 heartbeater.cc:464] Registering TS with master...
I20240501 08:02:58.642884 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 08:02:58.642941 21853 heartbeater.cc:511] Master 127.20.151.62:34829 requested a full tablet report, sending...
W20240501 08:02:58.643150 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:02:58.643365 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550578643350 us; error 0 us; skew 500 ppm
I20240501 08:02:58.643925 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:02:58.644955 21651 ts_manager.cc:194] Registered new tserver with Master: 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959)
I20240501 08:02:58.646412 21084 webserver.cc:460] Webserver started at http://127.20.151.3:40361/ using document root <none> and password file <none>
I20240501 08:02:58.646646 21651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59130
I20240501 08:02:58.647009 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:02:58.647264 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:02:58.647538 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:02:58.648494 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-2-root/instance:
uuid: "e282e3e8c9274b739f70534606f48166"
format_stamp: "Formatted at 2024-05-01 08:02:58 on dist-test-slave-pwkb"
I20240501 08:02:58.652240 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 08:02:58.655072 21864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:58.655733 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 08:02:58.655982 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-2-root
uuid: "e282e3e8c9274b739f70534606f48166"
format_stamp: "Formatted at 2024-05-01 08:02:58 on dist-test-slave-pwkb"
I20240501 08:02:58.656240 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:02:58.666472 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:02:58.667281 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:02:58.668503 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:02:58.670521 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:02:58.670701 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:58.670938 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:02:58.671082 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:02:58.706792 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:36377
I20240501 08:02:58.706861 21926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:36377 every 8 connection(s)
I20240501 08:02:58.718662 21927 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34829
I20240501 08:02:58.718981 21927 heartbeater.cc:464] Registering TS with master...
I20240501 08:02:58.719642 21927 heartbeater.cc:511] Master 127.20.151.62:34829 requested a full tablet report, sending...
I20240501 08:02:58.721338 21651 ts_manager.cc:194] Registered new tserver with Master: e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:02:58.721809 21084 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012097941s
I20240501 08:02:58.722573 21651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59140
I20240501 08:02:59.858587 21084 test_util.cc:268] Using random seed: -364704636
I20240501 08:02:59.873071 21778 heartbeater.cc:503] Master 127.20.151.62:34829 was elected leader, sending a full tablet report...
I20240501 08:02:59.898837 21927 heartbeater.cc:503] Master 127.20.151.62:34829 was elected leader, sending a full tablet report...
I20240501 08:02:59.904951 21853 heartbeater.cc:503] Master 127.20.151.62:34829 was elected leader, sending a full tablet report...
I20240501 08:02:59.925870 21651 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58848:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:02:59.928169 21651 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:02:59.999349 21892 tablet_service.cc:1467] Processing CreateTablet for tablet 0480d951fd9d40789d390a0871b46c02 (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 08:02:59.999349 21890 tablet_service.cc:1467] Processing CreateTablet for tablet 26dc8e957a2747528356007dab573637 (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 08:02:59.999817 21743 tablet_service.cc:1467] Processing CreateTablet for tablet 0480d951fd9d40789d390a0871b46c02 (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 08:03:00.000061 21889 tablet_service.cc:1467] Processing CreateTablet for tablet 234853d87e2d47edbb639e603fae75bb (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 08:03:00.000941 21892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0480d951fd9d40789d390a0871b46c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.001075 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0480d951fd9d40789d390a0871b46c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.001627 21889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234853d87e2d47edbb639e603fae75bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.001310 21891 tablet_service.cc:1467] Processing CreateTablet for tablet b72589ed0fdb463daad0083d072f7e01 (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 08:03:00.002483 21890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26dc8e957a2747528356007dab573637. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.003263 21891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72589ed0fdb463daad0083d072f7e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.010903 21742 tablet_service.cc:1467] Processing CreateTablet for tablet b72589ed0fdb463daad0083d072f7e01 (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 08:03:00.012166 21742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72589ed0fdb463daad0083d072f7e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.017908 21888 tablet_service.cc:1467] Processing CreateTablet for tablet 99c49ce3d47344b49a81863aacc787b2 (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 08:03:00.019197 21888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c49ce3d47344b49a81863aacc787b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.019223 21741 tablet_service.cc:1467] Processing CreateTablet for tablet 26dc8e957a2747528356007dab573637 (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 08:03:00.020535 21741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26dc8e957a2747528356007dab573637. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.022799 21887 tablet_service.cc:1467] Processing CreateTablet for tablet 63c05c8273a34209a6078c89d59bd01d (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 08:03:00.026440 21886 tablet_service.cc:1467] Processing CreateTablet for tablet 44bf2a078f494657bceaca86c03dbbfc (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 08:03:00.030037 21885 tablet_service.cc:1467] Processing CreateTablet for tablet b5710bd5ed314c1c8794e04c25b48b3e (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 08:03:00.031229 21740 tablet_service.cc:1467] Processing CreateTablet for tablet 234853d87e2d47edbb639e603fae75bb (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 08:03:00.031754 21887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c05c8273a34209a6078c89d59bd01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.032546 21740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234853d87e2d47edbb639e603fae75bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.043221 21738 tablet_service.cc:1467] Processing CreateTablet for tablet 63c05c8273a34209a6078c89d59bd01d (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 08:03:00.044551 21738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c05c8273a34209a6078c89d59bd01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.048431 21739 tablet_service.cc:1467] Processing CreateTablet for tablet 99c49ce3d47344b49a81863aacc787b2 (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 08:03:00.049832 21739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c49ce3d47344b49a81863aacc787b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.051141 21737 tablet_service.cc:1467] Processing CreateTablet for tablet 44bf2a078f494657bceaca86c03dbbfc (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 08:03:00.052363 21737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bf2a078f494657bceaca86c03dbbfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.059639 21736 tablet_service.cc:1467] Processing CreateTablet for tablet b5710bd5ed314c1c8794e04c25b48b3e (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 08:03:00.061218 21736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5710bd5ed314c1c8794e04c25b48b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.070834 21886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bf2a078f494657bceaca86c03dbbfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.074432 21885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5710bd5ed314c1c8794e04c25b48b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.081272 21948 tablet_bootstrap.cc:492] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119: Bootstrap starting.
I20240501 08:03:00.087333 21948 tablet_bootstrap.cc:654] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.113445 21817 tablet_service.cc:1467] Processing CreateTablet for tablet b72589ed0fdb463daad0083d072f7e01 (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 08:03:00.114223 21815 tablet_service.cc:1467] Processing CreateTablet for tablet 234853d87e2d47edbb639e603fae75bb (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 08:03:00.115768 21818 tablet_service.cc:1467] Processing CreateTablet for tablet 0480d951fd9d40789d390a0871b46c02 (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 08:03:00.116256 21814 tablet_service.cc:1467] Processing CreateTablet for tablet 99c49ce3d47344b49a81863aacc787b2 (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 08:03:00.115927 21817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72589ed0fdb463daad0083d072f7e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.118618 21948 tablet_bootstrap.cc:492] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119: No bootstrap required, opened a new log
I20240501 08:03:00.118721 21813 tablet_service.cc:1467] Processing CreateTablet for tablet 63c05c8273a34209a6078c89d59bd01d (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 08:03:00.120123 21948 ts_tablet_manager.cc:1367] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119: Time spent bootstrapping tablet: real 0.039s user 0.019s sys 0.000s
I20240501 08:03:00.120644 21811 tablet_service.cc:1467] Processing CreateTablet for tablet b5710bd5ed314c1c8794e04c25b48b3e (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 08:03:00.121225 21947 tablet_bootstrap.cc:492] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166: Bootstrap starting.
I20240501 08:03:00.119618 21812 tablet_service.cc:1467] Processing CreateTablet for tablet 44bf2a078f494657bceaca86c03dbbfc (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 08:03:00.123183 21816 tablet_service.cc:1467] Processing CreateTablet for tablet 26dc8e957a2747528356007dab573637 (DEFAULT_TABLE table=test-workload [id=3fd8b83a153440a68cd8c0905cd396ea]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 08:03:00.123925 21811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5710bd5ed314c1c8794e04c25b48b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.124753 21818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0480d951fd9d40789d390a0871b46c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.125315 21813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c05c8273a34209a6078c89d59bd01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.126109 21816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26dc8e957a2747528356007dab573637. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.126796 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c49ce3d47344b49a81863aacc787b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.127650 21815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234853d87e2d47edbb639e603fae75bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.131817 21948 raft_consensus.cc:348] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.133666 21948 raft_consensus.cc:374] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.135056 21948 raft_consensus.cc:729] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.135751 21948 consensus_queue.cc:260] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.137252 21812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bf2a078f494657bceaca86c03dbbfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:00.145756 21948 ts_tablet_manager.cc:1398] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119: Time spent starting tablet: real 0.025s user 0.006s sys 0.003s
I20240501 08:03:00.147123 21948 tablet_bootstrap.cc:492] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119: Bootstrap starting.
I20240501 08:03:00.152544 21948 tablet_bootstrap.cc:654] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.179159 21948 tablet_bootstrap.cc:492] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119: No bootstrap required, opened a new log
I20240501 08:03:00.179179 21947 tablet_bootstrap.cc:654] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.179863 21948 ts_tablet_manager.cc:1367] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.008s
I20240501 08:03:00.183576 21952 tablet_bootstrap.cc:492] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f: Bootstrap starting.
I20240501 08:03:00.184480 21948 raft_consensus.cc:348] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.185226 21948 raft_consensus.cc:374] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.185521 21948 raft_consensus.cc:729] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.192435 21952 tablet_bootstrap.cc:654] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.190128 21948 consensus_queue.cc:260] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.193321 21947 tablet_bootstrap.cc:492] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166: No bootstrap required, opened a new log
I20240501 08:03:00.193811 21947 ts_tablet_manager.cc:1367] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166: Time spent bootstrapping tablet: real 0.073s user 0.011s sys 0.007s
I20240501 08:03:00.195092 21948 ts_tablet_manager.cc:1398] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240501 08:03:00.195953 21948 tablet_bootstrap.cc:492] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119: Bootstrap starting.
I20240501 08:03:00.196420 21947 raft_consensus.cc:348] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.197175 21947 raft_consensus.cc:374] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.197484 21947 raft_consensus.cc:729] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282e3e8c9274b739f70534606f48166, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.198304 21947 consensus_queue.cc:260] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.202342 21948 tablet_bootstrap.cc:654] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.213393 21947 ts_tablet_manager.cc:1398] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166: Time spent starting tablet: real 0.019s user 0.001s sys 0.010s
I20240501 08:03:00.214725 21947 tablet_bootstrap.cc:492] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166: Bootstrap starting.
I20240501 08:03:00.221547 21947 tablet_bootstrap.cc:654] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.224463 21948 tablet_bootstrap.cc:492] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119: No bootstrap required, opened a new log
I20240501 08:03:00.224937 21948 ts_tablet_manager.cc:1367] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.011s
I20240501 08:03:00.226091 21952 tablet_bootstrap.cc:492] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f: No bootstrap required, opened a new log
I20240501 08:03:00.226512 21952 ts_tablet_manager.cc:1367] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f: Time spent bootstrapping tablet: real 0.043s user 0.018s sys 0.022s
I20240501 08:03:00.227646 21947 tablet_bootstrap.cc:492] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166: No bootstrap required, opened a new log
I20240501 08:03:00.227360 21948 raft_consensus.cc:348] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.228070 21947 ts_tablet_manager.cc:1367] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20240501 08:03:00.228101 21948 raft_consensus.cc:374] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.228287 21952 raft_consensus.cc:348] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.228624 21948 raft_consensus.cc:729] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.229001 21952 raft_consensus.cc:374] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.229317 21952 raft_consensus.cc:729] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab12efc498b426697bb1a5531064c2f, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.229461 21948 consensus_queue.cc:260] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.230038 21952 consensus_queue.cc:260] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.230333 21947 raft_consensus.cc:348] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } }
I20240501 08:03:00.231020 21947 raft_consensus.cc:374] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.231293 21947 raft_consensus.cc:729] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282e3e8c9274b739f70534606f48166, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.231873 21948 ts_tablet_manager.cc:1398] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240501 08:03:00.231992 21947 consensus_queue.cc:260] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } }
I20240501 08:03:00.233165 21948 tablet_bootstrap.cc:492] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119: Bootstrap starting.
I20240501 08:03:00.241495 21948 tablet_bootstrap.cc:654] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.241501 21947 ts_tablet_manager.cc:1398] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20240501 08:03:00.246655 21952 ts_tablet_manager.cc:1398] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
I20240501 08:03:00.247872 21952 tablet_bootstrap.cc:492] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f: Bootstrap starting.
I20240501 08:03:00.248327 21947 tablet_bootstrap.cc:492] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166: Bootstrap starting.
I20240501 08:03:00.253844 21947 tablet_bootstrap.cc:654] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.254586 21952 tablet_bootstrap.cc:654] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.261070 21950 raft_consensus.cc:482] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:00.261744 21950 raft_consensus.cc:504] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.263756 21947 tablet_bootstrap.cc:492] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166: No bootstrap required, opened a new log
I20240501 08:03:00.264125 21947 ts_tablet_manager.cc:1367] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20240501 08:03:00.266352 21952 tablet_bootstrap.cc:492] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f: No bootstrap required, opened a new log
I20240501 08:03:00.266852 21952 ts_tablet_manager.cc:1367] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.008s
I20240501 08:03:00.266659 21947 raft_consensus.cc:348] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.267634 21947 raft_consensus.cc:374] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.267993 21947 raft_consensus.cc:729] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282e3e8c9274b739f70534606f48166, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.268867 21947 consensus_queue.cc:260] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.270071 21952 raft_consensus.cc:348] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.271036 21952 raft_consensus.cc:374] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.271391 21952 raft_consensus.cc:729] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab12efc498b426697bb1a5531064c2f, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.266831 21950 leader_election.cc:288] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:00.261082 21948 tablet_bootstrap.cc:492] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119: No bootstrap required, opened a new log
I20240501 08:03:00.272699 21948 ts_tablet_manager.cc:1367] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.015s
I20240501 08:03:00.272449 21952 consensus_queue.cc:260] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.274354 21947 ts_tablet_manager.cc:1398] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240501 08:03:00.274479 21952 ts_tablet_manager.cc:1398] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:03:00.275290 21947 tablet_bootstrap.cc:492] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166: Bootstrap starting.
I20240501 08:03:00.275045 21948 raft_consensus.cc:348] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } }
I20240501 08:03:00.275380 21952 tablet_bootstrap.cc:492] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f: Bootstrap starting.
I20240501 08:03:00.275774 21948 raft_consensus.cc:374] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.281507 21952 tablet_bootstrap.cc:654] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.281502 21948 raft_consensus.cc:729] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.282354 21948 consensus_queue.cc:260] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } }
I20240501 08:03:00.288611 21947 tablet_bootstrap.cc:654] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.290004 21948 ts_tablet_manager.cc:1398] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20240501 08:03:00.293009 21952 tablet_bootstrap.cc:492] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f: No bootstrap required, opened a new log
I20240501 08:03:00.293466 21952 ts_tablet_manager.cc:1367] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20240501 08:03:00.294128 21948 tablet_bootstrap.cc:492] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119: Bootstrap starting.
I20240501 08:03:00.295840 21952 raft_consensus.cc:348] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.296582 21952 raft_consensus.cc:374] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.296872 21952 raft_consensus.cc:729] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab12efc498b426697bb1a5531064c2f, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.297636 21952 consensus_queue.cc:260] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.299710 21952 ts_tablet_manager.cc:1398] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:03:00.300745 21952 tablet_bootstrap.cc:492] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f: Bootstrap starting.
I20240501 08:03:00.303751 21948 tablet_bootstrap.cc:654] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.305778 21952 tablet_bootstrap.cc:654] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.311488 21947 tablet_bootstrap.cc:492] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166: No bootstrap required, opened a new log
I20240501 08:03:00.311921 21947 ts_tablet_manager.cc:1367] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.011s
I20240501 08:03:00.314460 21947 raft_consensus.cc:348] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.315173 21947 raft_consensus.cc:374] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.315495 21947 raft_consensus.cc:729] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282e3e8c9274b739f70534606f48166, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.316246 21947 consensus_queue.cc:260] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.316774 21828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234853d87e2d47edbb639e603fae75bb" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f" is_pre_election: true
I20240501 08:03:00.317510 21828 raft_consensus.cc:2454] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 0.
I20240501 08:03:00.319607 21948 tablet_bootstrap.cc:492] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119: No bootstrap required, opened a new log
I20240501 08:03:00.319906 21952 tablet_bootstrap.cc:492] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f: No bootstrap required, opened a new log
I20240501 08:03:00.320096 21948 ts_tablet_manager.cc:1367] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.007s
I20240501 08:03:00.320430 21952 ts_tablet_manager.cc:1367] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.002s
I20240501 08:03:00.322108 21947 ts_tablet_manager.cc:1398] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20240501 08:03:00.322590 21948 raft_consensus.cc:348] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.323045 21947 tablet_bootstrap.cc:492] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166: Bootstrap starting.
I20240501 08:03:00.322825 21952 raft_consensus.cc:348] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } }
I20240501 08:03:00.323683 21952 raft_consensus.cc:374] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.324018 21952 raft_consensus.cc:729] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab12efc498b426697bb1a5531064c2f, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.324838 21952 consensus_queue.cc:260] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } }
I20240501 08:03:00.328709 21952 ts_tablet_manager.cc:1398] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:03:00.328900 21947 tablet_bootstrap.cc:654] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.323350 21948 raft_consensus.cc:374] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.329753 21952 tablet_bootstrap.cc:492] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f: Bootstrap starting.
I20240501 08:03:00.329982 21948 raft_consensus.cc:729] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.330781 21948 consensus_queue.cc:260] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.335176 21948 ts_tablet_manager.cc:1398] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20240501 08:03:00.335580 21952 tablet_bootstrap.cc:654] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.336158 21948 tablet_bootstrap.cc:492] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119: Bootstrap starting.
I20240501 08:03:00.344805 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234853d87e2d47edbb639e603fae75bb" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166" is_pre_election: true
I20240501 08:03:00.345702 21902 raft_consensus.cc:2454] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 0.
I20240501 08:03:00.347388 21718 leader_election.cc:302] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, e282e3e8c9274b739f70534606f48166; no voters:
I20240501 08:03:00.348556 21950 raft_consensus.cc:2781] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:03:00.348987 21950 raft_consensus.cc:482] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:03:00.349391 21950 raft_consensus.cc:3037] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.358866 21950 raft_consensus.cc:504] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.361052 21947 tablet_bootstrap.cc:492] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166: No bootstrap required, opened a new log
I20240501 08:03:00.361513 21947 ts_tablet_manager.cc:1367] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.023s
I20240501 08:03:00.361750 21828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234853d87e2d47edbb639e603fae75bb" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f"
I20240501 08:03:00.362066 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234853d87e2d47edbb639e603fae75bb" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166"
I20240501 08:03:00.362838 21902 raft_consensus.cc:3037] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.363727 21947 raft_consensus.cc:348] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.364362 21947 raft_consensus.cc:374] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.364646 21947 raft_consensus.cc:729] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282e3e8c9274b739f70534606f48166, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.362367 21828 raft_consensus.cc:3037] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.365363 21947 consensus_queue.cc:260] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.367506 21947 ts_tablet_manager.cc:1398] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:03:00.368404 21947 tablet_bootstrap.cc:492] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166: Bootstrap starting.
I20240501 08:03:00.373862 21947 tablet_bootstrap.cc:654] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.378650 21955 raft_consensus.cc:482] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:00.379093 21955 raft_consensus.cc:504] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.379588 21902 raft_consensus.cc:2454] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 1.
I20240501 08:03:00.380949 21718 leader_election.cc:302] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, e282e3e8c9274b739f70534606f48166; no voters:
I20240501 08:03:00.381525 21955 leader_election.cc:288] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279), 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959)
I20240501 08:03:00.381927 21828 raft_consensus.cc:2454] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 1.
I20240501 08:03:00.382999 21950 leader_election.cc:288] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 election: Requested vote from peers 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:00.383755 21950 raft_consensus.cc:2781] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:00.384174 21950 raft_consensus.cc:686] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Becoming Leader. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Running, Role: LEADER
I20240501 08:03:00.384917 21950 consensus_queue.cc:237] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.391222 21948 tablet_bootstrap.cc:654] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.392462 21947 tablet_bootstrap.cc:492] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166: No bootstrap required, opened a new log
I20240501 08:03:00.393057 21947 ts_tablet_manager.cc:1367] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.005s
W20240501 08:03:00.394061 21699 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 08:03:00.395859 21704 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:00.396782 21704 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:00.396857 21952 tablet_bootstrap.cc:492] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f: No bootstrap required, opened a new log
I20240501 08:03:00.397560 21952 ts_tablet_manager.cc:1367] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f: Time spent bootstrapping tablet: real 0.068s user 0.026s sys 0.029s
I20240501 08:03:00.403148 21950 raft_consensus.cc:482] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:00.403827 21950 raft_consensus.cc:504] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.405051 21650 catalog_manager.cc:5484] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01d3577427624fa8b733111b0dc0d119 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "01d3577427624fa8b733111b0dc0d119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:00.406510 21828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c49ce3d47344b49a81863aacc787b2" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f" is_pre_election: true
I20240501 08:03:00.407197 21828 raft_consensus.cc:2454] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 0.
I20240501 08:03:00.407275 21947 raft_consensus.cc:348] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.408674 21716 leader_election.cc:302] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, 8ab12efc498b426697bb1a5531064c2f; no voters:
I20240501 08:03:00.412642 21947 raft_consensus.cc:374] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.412971 21947 raft_consensus.cc:729] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282e3e8c9274b739f70534606f48166, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.413779 21947 consensus_queue.cc:260] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.412039 21948 tablet_bootstrap.cc:492] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119: No bootstrap required, opened a new log
I20240501 08:03:00.412214 21971 raft_consensus.cc:482] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:00.414837 21948 ts_tablet_manager.cc:1367] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119: Time spent bootstrapping tablet: real 0.079s user 0.014s sys 0.000s
I20240501 08:03:00.414489 21952 raft_consensus.cc:348] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.415205 21952 raft_consensus.cc:374] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.415552 21952 raft_consensus.cc:729] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab12efc498b426697bb1a5531064c2f, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.416189 21952 consensus_queue.cc:260] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.414839 21971 raft_consensus.cc:504] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.418457 21950 leader_election.cc:288] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:00.419410 21950 raft_consensus.cc:2781] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:03:00.419831 21950 raft_consensus.cc:482] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:03:00.420200 21950 raft_consensus.cc:3037] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.420044 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c49ce3d47344b49a81863aacc787b2" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166" is_pre_election: true
I20240501 08:03:00.422801 21971 leader_election.cc:288] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:00.423231 21828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26dc8e957a2747528356007dab573637" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f" is_pre_election: true
I20240501 08:03:00.424539 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26dc8e957a2747528356007dab573637" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d3577427624fa8b733111b0dc0d119" is_pre_election: true
I20240501 08:03:00.424870 21947 ts_tablet_manager.cc:1398] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
W20240501 08:03:00.425037 21865 leader_election.cc:341] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:03:00.425184 21753 raft_consensus.cc:2454] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e282e3e8c9274b739f70534606f48166 in term 0.
I20240501 08:03:00.425902 21947 tablet_bootstrap.cc:492] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166: Bootstrap starting.
I20240501 08:03:00.426399 21868 leader_election.cc:302] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, e282e3e8c9274b739f70534606f48166; no voters: 8ab12efc498b426697bb1a5531064c2f
I20240501 08:03:00.426612 21828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bf2a078f494657bceaca86c03dbbfc" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f" is_pre_election: true
I20240501 08:03:00.427539 21955 raft_consensus.cc:2781] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:03:00.428179 21955 raft_consensus.cc:482] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 08:03:00.428377 21716 leader_election.cc:341] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:03:00.428561 21955 raft_consensus.cc:3037] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.429174 21950 raft_consensus.cc:504] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.431450 21947 tablet_bootstrap.cc:654] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.431823 21828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c49ce3d47344b49a81863aacc787b2" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f"
I20240501 08:03:00.432423 21828 raft_consensus.cc:3037] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.434829 21955 raft_consensus.cc:504] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.438071 21955 leader_election.cc:288] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 election: Requested vote from peers 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279), 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959)
I20240501 08:03:00.437952 21827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26dc8e957a2747528356007dab573637" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f"
I20240501 08:03:00.438928 21950 leader_election.cc:288] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 election: Requested vote from peers 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
W20240501 08:03:00.421756 21718 leader_election.cc:341] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377): Illegal state: must be running to vote when last-logged opid is not known
W20240501 08:03:00.439190 21865 leader_election.cc:341] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:03:00.442263 21828 raft_consensus.cc:2454] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 1.
I20240501 08:03:00.443388 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bf2a078f494657bceaca86c03dbbfc" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166" is_pre_election: true
I20240501 08:03:00.444007 21902 raft_consensus.cc:2454] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 0.
I20240501 08:03:00.443899 21716 leader_election.cc:302] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, 8ab12efc498b426697bb1a5531064c2f; no voters:
I20240501 08:03:00.444768 21950 raft_consensus.cc:2781] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:00.445246 21950 raft_consensus.cc:686] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Becoming Leader. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Running, Role: LEADER
I20240501 08:03:00.445439 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c49ce3d47344b49a81863aacc787b2" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166"
I20240501 08:03:00.445075 21718 leader_election.cc:302] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, e282e3e8c9274b739f70534606f48166; no voters: 8ab12efc498b426697bb1a5531064c2f
I20240501 08:03:00.446077 21950 consensus_queue.cc:237] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.418375 21948 raft_consensus.cc:348] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.439740 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26dc8e957a2747528356007dab573637" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d3577427624fa8b733111b0dc0d119"
I20240501 08:03:00.447176 21976 raft_consensus.cc:2781] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:03:00.447340 21753 raft_consensus.cc:3037] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.447552 21976 raft_consensus.cc:482] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:03:00.447903 21976 raft_consensus.cc:3037] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.448618 21947 tablet_bootstrap.cc:492] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166: No bootstrap required, opened a new log
I20240501 08:03:00.449084 21947 ts_tablet_manager.cc:1367] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.011s
I20240501 08:03:00.452006 21947 raft_consensus.cc:348] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.452893 21947 raft_consensus.cc:374] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.453238 21947 raft_consensus.cc:729] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282e3e8c9274b739f70534606f48166, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.441265 21952 ts_tablet_manager.cc:1398] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f: Time spent starting tablet: real 0.043s user 0.005s sys 0.000s
I20240501 08:03:00.453891 21976 raft_consensus.cc:504] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.454510 21952 tablet_bootstrap.cc:492] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f: Bootstrap starting.
I20240501 08:03:00.454135 21947 consensus_queue.cc:260] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.455757 21976 leader_election.cc:288] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 election: Requested vote from peers 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:00.456683 21947 ts_tablet_manager.cc:1398] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240501 08:03:00.457666 21947 tablet_bootstrap.cc:492] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166: Bootstrap starting.
I20240501 08:03:00.459367 21952 tablet_bootstrap.cc:654] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.459234 21828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bf2a078f494657bceaca86c03dbbfc" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f"
I20240501 08:03:00.460040 21753 raft_consensus.cc:2454] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e282e3e8c9274b739f70534606f48166 in term 1.
I20240501 08:03:00.461094 21868 leader_election.cc:302] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, e282e3e8c9274b739f70534606f48166; no voters: 8ab12efc498b426697bb1a5531064c2f
I20240501 08:03:00.462152 21955 raft_consensus.cc:2781] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:00.463049 21947 tablet_bootstrap.cc:654] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166: Neither blocks nor log segments found. Creating new log.
W20240501 08:03:00.463979 21716 leader_election.cc:341] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:03:00.468323 21948 raft_consensus.cc:374] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.468706 21948 raft_consensus.cc:729] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.469370 21955 raft_consensus.cc:686] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 1 LEADER]: Becoming Leader. State: Replica: e282e3e8c9274b739f70534606f48166, State: Running, Role: LEADER
I20240501 08:03:00.470413 21958 raft_consensus.cc:482] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:00.470281 21955 consensus_queue.cc:237] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.471068 21958 raft_consensus.cc:504] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.473104 21952 tablet_bootstrap.cc:492] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f: No bootstrap required, opened a new log
I20240501 08:03:00.473760 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bf2a078f494657bceaca86c03dbbfc" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166"
I20240501 08:03:00.474337 21902 raft_consensus.cc:3037] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:03:00.474871 21718 leader_election.cc:341] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:03:00.476002 21948 consensus_queue.cc:260] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.478529 21948 ts_tablet_manager.cc:1398] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119: Time spent starting tablet: real 0.063s user 0.007s sys 0.000s
I20240501 08:03:00.479439 21948 tablet_bootstrap.cc:492] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119: Bootstrap starting.
I20240501 08:03:00.482059 21902 raft_consensus.cc:2454] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 1.
I20240501 08:03:00.483309 21979 raft_consensus.cc:482] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:00.483044 21718 leader_election.cc:302] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, e282e3e8c9274b739f70534606f48166; no voters: 8ab12efc498b426697bb1a5531064c2f
I20240501 08:03:00.483781 21979 raft_consensus.cc:504] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.484331 21976 raft_consensus.cc:2781] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:00.484896 21958 leader_election.cc:288] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:00.483050 21952 ts_tablet_manager.cc:1367] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.001s
I20240501 08:03:00.489259 21952 raft_consensus.cc:348] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.489974 21952 raft_consensus.cc:374] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.490304 21952 raft_consensus.cc:729] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab12efc498b426697bb1a5531064c2f, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.490957 21952 consensus_queue.cc:260] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.493427 21979 leader_election.cc:288] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279), 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959)
I20240501 08:03:00.493372 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c05c8273a34209a6078c89d59bd01d" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d3577427624fa8b733111b0dc0d119" is_pre_election: true
W20240501 08:03:00.494822 21868 leader_election.cc:341] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:03:00.484874 21976 raft_consensus.cc:686] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Becoming Leader. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Running, Role: LEADER
I20240501 08:03:00.496167 21976 consensus_queue.cc:237] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.499030 21952 ts_tablet_manager.cc:1398] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240501 08:03:00.499925 21952 tablet_bootstrap.cc:492] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f: Bootstrap starting.
I20240501 08:03:00.502116 21947 tablet_bootstrap.cc:492] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166: No bootstrap required, opened a new log
I20240501 08:03:00.502763 21947 ts_tablet_manager.cc:1367] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.026s
I20240501 08:03:00.506776 21947 raft_consensus.cc:348] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.507477 21947 raft_consensus.cc:374] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.507882 21947 raft_consensus.cc:729] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282e3e8c9274b739f70534606f48166, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.508635 21947 consensus_queue.cc:260] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.509198 21650 catalog_manager.cc:5484] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 reported cstate change: term changed from 0 to 1, leader changed from <none> to e282e3e8c9274b739f70534606f48166 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "e282e3e8c9274b739f70534606f48166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } health_report { overall_health: HEALTHY } } }
I20240501 08:03:00.510950 21947 ts_tablet_manager.cc:1398] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240501 08:03:00.513108 21948 tablet_bootstrap.cc:654] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.519816 21828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c05c8273a34209a6078c89d59bd01d" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f" is_pre_election: true
I20240501 08:03:00.521157 21828 raft_consensus.cc:2454] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e282e3e8c9274b739f70534606f48166 in term 0.
I20240501 08:03:00.526897 21865 leader_election.cc:302] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ab12efc498b426697bb1a5531064c2f, e282e3e8c9274b739f70534606f48166; no voters: 01d3577427624fa8b733111b0dc0d119
I20240501 08:03:00.528250 21971 raft_consensus.cc:482] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:00.528705 21971 raft_consensus.cc:504] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } }
I20240501 08:03:00.530398 21955 raft_consensus.cc:2781] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:03:00.530649 21971 leader_election.cc:288] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377), 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959)
I20240501 08:03:00.530772 21955 raft_consensus.cc:482] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:03:00.531121 21955 raft_consensus.cc:3037] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.532339 21828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0480d951fd9d40789d390a0871b46c02" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f" is_pre_election: true
I20240501 08:03:00.532935 21828 raft_consensus.cc:2454] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 0.
I20240501 08:03:00.533887 21716 leader_election.cc:302] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, 8ab12efc498b426697bb1a5531064c2f; no voters:
I20240501 08:03:00.534679 21950 raft_consensus.cc:2781] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:03:00.534711 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0480d951fd9d40789d390a0871b46c02" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166" is_pre_election: true
I20240501 08:03:00.535763 21902 raft_consensus.cc:2454] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 0.
I20240501 08:03:00.535135 21950 raft_consensus.cc:482] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:03:00.536798 21948 tablet_bootstrap.cc:492] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119: No bootstrap required, opened a new log
I20240501 08:03:00.537166 21950 raft_consensus.cc:3037] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.537362 21948 ts_tablet_manager.cc:1367] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.002s
I20240501 08:03:00.537312 21649 catalog_manager.cc:5484] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01d3577427624fa8b733111b0dc0d119 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "01d3577427624fa8b733111b0dc0d119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:00.538759 21955 raft_consensus.cc:504] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.539945 21948 raft_consensus.cc:348] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.540645 21948 raft_consensus.cc:374] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.540907 21948 raft_consensus.cc:729] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.541234 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c05c8273a34209a6078c89d59bd01d" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d3577427624fa8b733111b0dc0d119"
I20240501 08:03:00.541584 21948 consensus_queue.cc:260] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.542318 21979 raft_consensus.cc:482] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:00.542753 21979 raft_consensus.cc:504] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.543884 21948 ts_tablet_manager.cc:1398] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240501 08:03:00.543774 21950 raft_consensus.cc:504] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } }
I20240501 08:03:00.544512 21979 leader_election.cc:288] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279), 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959)
W20240501 08:03:00.545990 21868 leader_election.cc:341] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:03:00.546440 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0480d951fd9d40789d390a0871b46c02" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166"
I20240501 08:03:00.546945 21952 tablet_bootstrap.cc:654] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.547003 21902 raft_consensus.cc:3037] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.547078 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72589ed0fdb463daad0083d072f7e01" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d3577427624fa8b733111b0dc0d119" is_pre_election: true
I20240501 08:03:00.548131 21753 raft_consensus.cc:2454] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e282e3e8c9274b739f70534606f48166 in term 0.
I20240501 08:03:00.549180 21868 leader_election.cc:302] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, e282e3e8c9274b739f70534606f48166; no voters:
I20240501 08:03:00.550015 21979 raft_consensus.cc:2781] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:03:00.550323 21979 raft_consensus.cc:482] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:03:00.550643 21979 raft_consensus.cc:3037] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.551081 21955 leader_election.cc:288] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 election: Requested vote from peers 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279), 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959)
I20240501 08:03:00.551295 21827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c05c8273a34209a6078c89d59bd01d" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f"
I20240501 08:03:00.551945 21827 raft_consensus.cc:3037] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.552861 21902 raft_consensus.cc:2454] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 1.
I20240501 08:03:00.554463 21948 tablet_bootstrap.cc:492] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119: Bootstrap starting.
I20240501 08:03:00.557830 21718 leader_election.cc:302] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, e282e3e8c9274b739f70534606f48166; no voters:
I20240501 08:03:00.558148 21828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72589ed0fdb463daad0083d072f7e01" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f" is_pre_election: true
I20240501 08:03:00.559044 21828 raft_consensus.cc:2454] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e282e3e8c9274b739f70534606f48166 in term 0.
I20240501 08:03:00.559435 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72589ed0fdb463daad0083d072f7e01" candidate_uuid: "8ab12efc498b426697bb1a5531064c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166" is_pre_election: true
I20240501 08:03:00.561143 21948 tablet_bootstrap.cc:654] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.561676 21826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0480d951fd9d40789d390a0871b46c02" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f"
I20240501 08:03:00.562337 21826 raft_consensus.cc:3037] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.564028 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72589ed0fdb463daad0083d072f7e01" candidate_uuid: "8ab12efc498b426697bb1a5531064c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d3577427624fa8b733111b0dc0d119" is_pre_election: true
I20240501 08:03:00.564769 21753 raft_consensus.cc:2454] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ab12efc498b426697bb1a5531064c2f in term 0.
I20240501 08:03:00.565949 21794 leader_election.cc:302] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, 8ab12efc498b426697bb1a5531064c2f; no voters:
I20240501 08:03:00.566084 21971 raft_consensus.cc:2781] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:00.566818 21958 raft_consensus.cc:2781] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:03:00.567179 21958 raft_consensus.cc:482] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:03:00.567499 21958 raft_consensus.cc:3037] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.569975 21826 raft_consensus.cc:2454] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 1.
I20240501 08:03:00.571322 21950 leader_election.cc:288] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 election: Requested vote from peers e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377), 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959)
I20240501 08:03:00.574239 21958 raft_consensus.cc:504] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
W20240501 08:03:00.577260 21774 debug-util.cc:398] Leaking SignalData structure 0x7b08000a57c0 after lost signal to thread 21087
I20240501 08:03:00.577518 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72589ed0fdb463daad0083d072f7e01" candidate_uuid: "8ab12efc498b426697bb1a5531064c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d3577427624fa8b733111b0dc0d119"
I20240501 08:03:00.578032 21753 raft_consensus.cc:3037] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.583937 21753 raft_consensus.cc:2454] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ab12efc498b426697bb1a5531064c2f in term 1.
I20240501 08:03:00.585127 21794 leader_election.cc:302] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, 8ab12efc498b426697bb1a5531064c2f; no voters:
I20240501 08:03:00.590140 21958 leader_election.cc:288] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [CANDIDATE]: Term 1 election: Requested vote from peers 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:00.590976 21958 raft_consensus.cc:2781] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:00.596571 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72589ed0fdb463daad0083d072f7e01" candidate_uuid: "8ab12efc498b426697bb1a5531064c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166"
I20240501 08:03:00.600840 21971 raft_consensus.cc:686] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Becoming Leader. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Running, Role: LEADER
I20240501 08:03:00.597019 21958 raft_consensus.cc:686] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 1 LEADER]: Becoming Leader. State: Replica: 8ab12efc498b426697bb1a5531064c2f, State: Running, Role: LEADER
I20240501 08:03:00.605736 21958 consensus_queue.cc:237] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.605538 21971 consensus_queue.cc:237] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } }
I20240501 08:03:00.606307 21979 raft_consensus.cc:504] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.609651 21955 raft_consensus.cc:482] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:00.610824 21902 raft_consensus.cc:2379] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ab12efc498b426697bb1a5531064c2f in current term 1: Already voted for candidate e282e3e8c9274b739f70534606f48166 in this term.
I20240501 08:03:00.610446 21955 raft_consensus.cc:504] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.604708 21827 raft_consensus.cc:2454] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e282e3e8c9274b739f70534606f48166 in term 1.
I20240501 08:03:00.615324 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72589ed0fdb463daad0083d072f7e01" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d3577427624fa8b733111b0dc0d119"
I20240501 08:03:00.615967 21955 leader_election.cc:288] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279), 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959)
I20240501 08:03:00.616117 21753 raft_consensus.cc:2379] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e282e3e8c9274b739f70534606f48166 in current term 1: Already voted for candidate 8ab12efc498b426697bb1a5531064c2f in this term.
I20240501 08:03:00.616434 21865 leader_election.cc:302] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ab12efc498b426697bb1a5531064c2f, e282e3e8c9274b739f70534606f48166; no voters: 01d3577427624fa8b733111b0dc0d119
I20240501 08:03:00.617066 21752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c49ce3d47344b49a81863aacc787b2" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d3577427624fa8b733111b0dc0d119" is_pre_election: true
I20240501 08:03:00.618391 21955 raft_consensus.cc:2781] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:00.619594 21952 tablet_bootstrap.cc:492] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f: No bootstrap required, opened a new log
I20240501 08:03:00.619994 21979 leader_election.cc:288] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 election: Requested vote from peers 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279), 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959)
I20240501 08:03:00.620159 21952 ts_tablet_manager.cc:1367] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f: Time spent bootstrapping tablet: real 0.120s user 0.019s sys 0.042s
I20240501 08:03:00.620585 21828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72589ed0fdb463daad0083d072f7e01" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f"
I20240501 08:03:00.622669 21865 leader_election.cc:302] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e282e3e8c9274b739f70534606f48166; no voters: 01d3577427624fa8b733111b0dc0d119, 8ab12efc498b426697bb1a5531064c2f
I20240501 08:03:00.624116 21991 raft_consensus.cc:2726] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:03:00.622323 21955 raft_consensus.cc:686] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 1 LEADER]: Becoming Leader. State: Replica: e282e3e8c9274b739f70534606f48166, State: Running, Role: LEADER
I20240501 08:03:00.623567 21827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c49ce3d47344b49a81863aacc787b2" candidate_uuid: "e282e3e8c9274b739f70534606f48166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f" is_pre_election: true
I20240501 08:03:00.624838 21948 tablet_bootstrap.cc:492] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119: No bootstrap required, opened a new log
I20240501 08:03:00.625460 21948 ts_tablet_manager.cc:1367] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119: Time spent bootstrapping tablet: real 0.071s user 0.019s sys 0.005s
I20240501 08:03:00.625823 21955 consensus_queue.cc:237] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.626082 21952 raft_consensus.cc:348] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.627044 21952 raft_consensus.cc:374] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.627331 21952 raft_consensus.cc:729] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab12efc498b426697bb1a5531064c2f, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.628012 21948 raft_consensus.cc:348] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.628055 21952 consensus_queue.cc:260] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.628808 21948 raft_consensus.cc:374] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.629097 21948 raft_consensus.cc:729] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.629841 21948 consensus_queue.cc:260] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.637986 21948 ts_tablet_manager.cc:1398] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240501 08:03:00.638041 21952 ts_tablet_manager.cc:1398] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f: Time spent starting tablet: real 0.017s user 0.006s sys 0.003s
I20240501 08:03:00.639609 21952 tablet_bootstrap.cc:492] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f: Bootstrap starting.
I20240501 08:03:00.637214 21827 raft_consensus.cc:2379] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e282e3e8c9274b739f70534606f48166 in current term 1: Already voted for candidate 01d3577427624fa8b733111b0dc0d119 in this term.
I20240501 08:03:00.644915 21865 leader_election.cc:302] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e282e3e8c9274b739f70534606f48166; no voters: 01d3577427624fa8b733111b0dc0d119, 8ab12efc498b426697bb1a5531064c2f
I20240501 08:03:00.646173 21952 tablet_bootstrap.cc:654] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:00.646306 21991 raft_consensus.cc:3037] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.643522 21971 raft_consensus.cc:482] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:00.648106 21649 catalog_manager.cc:5484] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01d3577427624fa8b733111b0dc0d119 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "01d3577427624fa8b733111b0dc0d119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:00.654418 21991 raft_consensus.cc:2726] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:03:00.655460 21649 catalog_manager.cc:5484] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01d3577427624fa8b733111b0dc0d119 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "01d3577427624fa8b733111b0dc0d119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:00.657472 21650 catalog_manager.cc:5484] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 reported cstate change: term changed from 0 to 1, leader changed from <none> to e282e3e8c9274b739f70534606f48166 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "e282e3e8c9274b739f70534606f48166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } health_report { overall_health: HEALTHY } } }
I20240501 08:03:00.665783 21971 raft_consensus.cc:504] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.669718 21827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5710bd5ed314c1c8794e04c25b48b3e" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f" is_pre_election: true
I20240501 08:03:00.670523 21827 raft_consensus.cc:2454] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 0.
I20240501 08:03:00.671840 21716 leader_election.cc:302] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, 8ab12efc498b426697bb1a5531064c2f; no voters:
I20240501 08:03:00.673069 21989 raft_consensus.cc:2781] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:03:00.673573 21989 raft_consensus.cc:482] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:03:00.674008 21989 raft_consensus.cc:3037] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.684474 21989 raft_consensus.cc:504] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.686926 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5710bd5ed314c1c8794e04c25b48b3e" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166" is_pre_election: true
I20240501 08:03:00.687819 21902 raft_consensus.cc:2454] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 0.
I20240501 08:03:00.688438 21989 leader_election.cc:288] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 election: Requested vote from peers 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:00.689797 21827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5710bd5ed314c1c8794e04c25b48b3e" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f"
I20240501 08:03:00.690333 21827 raft_consensus.cc:3037] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.690798 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5710bd5ed314c1c8794e04c25b48b3e" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166"
I20240501 08:03:00.691346 21902 raft_consensus.cc:3037] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.683876 21651 catalog_manager.cc:5484] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ab12efc498b426697bb1a5531064c2f (127.20.151.2). New cstate: current_term: 1 leader_uuid: "8ab12efc498b426697bb1a5531064c2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:00.692427 21952 tablet_bootstrap.cc:492] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f: No bootstrap required, opened a new log
I20240501 08:03:00.686062 21971 leader_election.cc:288] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:00.693111 21952 ts_tablet_manager.cc:1367] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f: Time spent bootstrapping tablet: real 0.054s user 0.016s sys 0.034s
I20240501 08:03:00.696650 21827 raft_consensus.cc:2454] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 1.
I20240501 08:03:00.696823 21952 raft_consensus.cc:348] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.697836 21952 raft_consensus.cc:374] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:00.697651 21716 leader_election.cc:302] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, 8ab12efc498b426697bb1a5531064c2f; no voters:
I20240501 08:03:00.698323 21952 raft_consensus.cc:729] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab12efc498b426697bb1a5531064c2f, State: Initialized, Role: FOLLOWER
I20240501 08:03:00.698640 21971 raft_consensus.cc:2781] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:00.699079 21971 raft_consensus.cc:686] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Becoming Leader. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Running, Role: LEADER
I20240501 08:03:00.699479 21989 raft_consensus.cc:482] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:00.699040 21952 consensus_queue.cc:260] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.699880 21989 raft_consensus.cc:504] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.699982 21971 consensus_queue.cc:237] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:00.701267 21952 ts_tablet_manager.cc:1398] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240501 08:03:00.701815 21989 leader_election.cc:288] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ab12efc498b426697bb1a5531064c2f (127.20.151.2:44959), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:00.702807 21827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c05c8273a34209a6078c89d59bd01d" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab12efc498b426697bb1a5531064c2f" is_pre_election: true
I20240501 08:03:00.703094 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c05c8273a34209a6078c89d59bd01d" candidate_uuid: "01d3577427624fa8b733111b0dc0d119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282e3e8c9274b739f70534606f48166" is_pre_election: true
I20240501 08:03:00.703768 21827 raft_consensus.cc:2379] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01d3577427624fa8b733111b0dc0d119 in current term 1: Already voted for candidate e282e3e8c9274b739f70534606f48166 in this term.
I20240501 08:03:00.704766 21716 leader_election.cc:302] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119; no voters: 8ab12efc498b426697bb1a5531064c2f, e282e3e8c9274b739f70534606f48166
I20240501 08:03:00.706073 21989 raft_consensus.cc:3037] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.706094 21902 raft_consensus.cc:2454] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d3577427624fa8b733111b0dc0d119 in term 1.
I20240501 08:03:00.712191 21989 raft_consensus.cc:2726] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:03:00.716863 21651 catalog_manager.cc:5484] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01d3577427624fa8b733111b0dc0d119 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "01d3577427624fa8b733111b0dc0d119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:00.782742 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:00.790472 21997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:00.791976 21998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:00.795224 22000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:00.796197 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:00.797798 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:00.798091 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:00.798303 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550580798286 us; error 0 us; skew 500 ppm
I20240501 08:03:00.799252 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:00.802336 21084 webserver.cc:460] Webserver started at http://127.20.151.4:42975/ using document root <none> and password file <none>
I20240501 08:03:00.802956 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:00.803186 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:00.803507 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:00.805083 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-3-root/instance:
uuid: "568380ed8e1a4956a94c0cad49fc4a0d"
format_stamp: "Formatted at 2024-05-01 08:03:00 on dist-test-slave-pwkb"
I20240501 08:03:00.811278 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:03:00.815550 22005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:00.816345 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 08:03:00.816614 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-3-root
uuid: "568380ed8e1a4956a94c0cad49fc4a0d"
format_stamp: "Formatted at 2024-05-01 08:03:00 on dist-test-slave-pwkb"
I20240501 08:03:00.816880 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550573434241-21084-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:00.839197 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:00.840159 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:00.841583 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:03:00.844074 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:03:00.844285 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:00.844511 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:03:00.844659 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:00.882516 21827 raft_consensus.cc:3037] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:00.892568 21991 consensus_queue.cc:1035] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:00.920271 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.4:39747
I20240501 08:03:00.920938 21989 consensus_queue.cc:1035] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:00.937896 21991 consensus_queue.cc:1035] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:00.945564 21971 consensus_queue.cc:1035] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:00.920382 22070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.4:39747 every 8 connection(s)
I20240501 08:03:00.962085 22071 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34829
I20240501 08:03:00.962544 22071 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:00.963634 22071 heartbeater.cc:511] Master 127.20.151.62:34829 requested a full tablet report, sending...
I20240501 08:03:00.966667 21651 ts_manager.cc:194] Registered new tserver with Master: 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4:39747)
I20240501 08:03:00.968735 21651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58850
I20240501 08:03:00.978789 21971 consensus_queue.cc:1035] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:03:00.987481 21989 consensus_queue.cc:1035] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:03:01.005095 21827 raft_consensus.cc:3037] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:01.018330 21976 consensus_queue.cc:1035] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:03:01.023274 21950 consensus_queue.cc:1035] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:01.121778 21958 consensus_queue.cc:1035] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:01.131366 21976 consensus_queue.cc:1035] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:01.139894 21958 consensus_queue.cc:1035] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:01.152895 21991 consensus_queue.cc:1035] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:01.161864 21989 consensus_queue.cc:1035] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:01.181813 21979 consensus_queue.cc:1035] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:01.230971 21976 consensus_queue.cc:1035] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:01.240691 21971 consensus_queue.cc:1035] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:01.971211 22071 heartbeater.cc:503] Master 127.20.151.62:34829 was elected leader, sending a full tablet report...
I20240501 08:03:02.397915 21704 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:02.415966 21753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "234853d87e2d47edbb639e603fae75bb"
dest_uuid: "01d3577427624fa8b733111b0dc0d119"
mode: GRACEFUL
new_leader_uuid: "8ab12efc498b426697bb1a5531064c2f"
from {username='slave'} at 127.0.0.1:58368
I20240501 08:03:02.416548 21753 raft_consensus.cc:595] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Received request to transfer leadership to TS 8ab12efc498b426697bb1a5531064c2f
I20240501 08:03:02.418900 21753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99c49ce3d47344b49a81863aacc787b2"
dest_uuid: "01d3577427624fa8b733111b0dc0d119"
mode: GRACEFUL
new_leader_uuid: "8ab12efc498b426697bb1a5531064c2f"
from {username='slave'} at 127.0.0.1:58368
I20240501 08:03:02.419366 21753 raft_consensus.cc:595] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Received request to transfer leadership to TS 8ab12efc498b426697bb1a5531064c2f
I20240501 08:03:02.420246 21704 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 08:03:02.420683 21704 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:02.465246 22097 raft_consensus.cc:979] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119: : Instructing follower 8ab12efc498b426697bb1a5531064c2f to start an election
I20240501 08:03:02.465713 21971 raft_consensus.cc:1067] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Signalling peer 8ab12efc498b426697bb1a5531064c2f to start an election
I20240501 08:03:02.467348 21828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "234853d87e2d47edbb639e603fae75bb"
dest_uuid: "8ab12efc498b426697bb1a5531064c2f"
from {username='slave'} at 127.0.0.1:35340
I20240501 08:03:02.467986 21828 raft_consensus.cc:482] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:03:02.468405 21828 raft_consensus.cc:3037] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:03:02.475402 21828 raft_consensus.cc:504] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:02.477923 21828 leader_election.cc:288] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [CANDIDATE]: Term 2 election: Requested vote from peers 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:02.478502 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234853d87e2d47edbb639e603fae75bb" candidate_uuid: "8ab12efc498b426697bb1a5531064c2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e282e3e8c9274b739f70534606f48166"
I20240501 08:03:02.478204 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234853d87e2d47edbb639e603fae75bb" candidate_uuid: "8ab12efc498b426697bb1a5531064c2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "01d3577427624fa8b733111b0dc0d119"
I20240501 08:03:02.479096 21902 raft_consensus.cc:3037] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:03:02.479223 21753 raft_consensus.cc:3032] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:03:02.479584 21753 raft_consensus.cc:729] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Running, Role: LEADER
I20240501 08:03:02.480314 21753 consensus_queue.cc:260] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:02.481499 21753 raft_consensus.cc:3037] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:03:02.485186 21902 raft_consensus.cc:2454] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ab12efc498b426697bb1a5531064c2f in term 2.
I20240501 08:03:02.486284 21793 leader_election.cc:302] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ab12efc498b426697bb1a5531064c2f, e282e3e8c9274b739f70534606f48166; no voters:
I20240501 08:03:02.487175 22086 raft_consensus.cc:2781] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:03:02.489410 22097 raft_consensus.cc:979] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119: : Instructing follower 8ab12efc498b426697bb1a5531064c2f to start an election
I20240501 08:03:02.489965 21971 raft_consensus.cc:1067] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Signalling peer 8ab12efc498b426697bb1a5531064c2f to start an election
I20240501 08:03:02.490358 22086 raft_consensus.cc:686] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 2 LEADER]: Becoming Leader. State: Replica: 8ab12efc498b426697bb1a5531064c2f, State: Running, Role: LEADER
I20240501 08:03:02.490876 21753 raft_consensus.cc:2454] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ab12efc498b426697bb1a5531064c2f in term 2.
I20240501 08:03:02.491544 21828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99c49ce3d47344b49a81863aacc787b2"
dest_uuid: "8ab12efc498b426697bb1a5531064c2f"
from {username='slave'} at 127.0.0.1:35340
I20240501 08:03:02.491360 22086 consensus_queue.cc:237] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:02.492063 21828 raft_consensus.cc:482] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:03:02.492444 21828 raft_consensus.cc:3037] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:03:02.497498 21828 raft_consensus.cc:504] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:02.499205 21828 leader_election.cc:288] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [CANDIDATE]: Term 2 election: Requested vote from peers 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279), e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:02.499918 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c49ce3d47344b49a81863aacc787b2" candidate_uuid: "8ab12efc498b426697bb1a5531064c2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "01d3577427624fa8b733111b0dc0d119"
I20240501 08:03:02.500543 21753 raft_consensus.cc:3032] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:03:02.500990 21753 raft_consensus.cc:729] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 01d3577427624fa8b733111b0dc0d119, State: Running, Role: LEADER
I20240501 08:03:02.500597 21650 catalog_manager.cc:5484] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f reported cstate change: term changed from 1 to 2, leader changed from 01d3577427624fa8b733111b0dc0d119 (127.20.151.1) to 8ab12efc498b426697bb1a5531064c2f (127.20.151.2). New cstate: current_term: 2 leader_uuid: "8ab12efc498b426697bb1a5531064c2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:02.501408 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c49ce3d47344b49a81863aacc787b2" candidate_uuid: "8ab12efc498b426697bb1a5531064c2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e282e3e8c9274b739f70534606f48166"
I20240501 08:03:02.502118 21902 raft_consensus.cc:3037] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:03:02.501789 21753 consensus_queue.cc:260] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:02.503144 21753 raft_consensus.cc:3037] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:03:02.509896 21753 raft_consensus.cc:2454] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ab12efc498b426697bb1a5531064c2f in term 2.
I20240501 08:03:02.510949 21794 leader_election.cc:302] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d3577427624fa8b733111b0dc0d119, 8ab12efc498b426697bb1a5531064c2f; no voters:
I20240501 08:03:02.510154 21902 raft_consensus.cc:2454] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ab12efc498b426697bb1a5531064c2f in term 2.
I20240501 08:03:02.511834 22086 raft_consensus.cc:2781] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:03:02.512513 22086 raft_consensus.cc:686] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 2 LEADER]: Becoming Leader. State: Replica: 8ab12efc498b426697bb1a5531064c2f, State: Running, Role: LEADER
I20240501 08:03:02.513288 22086 consensus_queue.cc:237] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } }
I20240501 08:03:02.526291 21650 catalog_manager.cc:5484] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f reported cstate change: term changed from 1 to 2, leader changed from 01d3577427624fa8b733111b0dc0d119 (127.20.151.1) to 8ab12efc498b426697bb1a5531064c2f (127.20.151.2). New cstate: current_term: 2 leader_uuid: "8ab12efc498b426697bb1a5531064c2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:02.904834 21902 raft_consensus.cc:1261] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [term 2 FOLLOWER]: Refusing update from remote peer 8ab12efc498b426697bb1a5531064c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:03:02.906157 22086 consensus_queue.cc:1035] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:03:02.916383 21753 raft_consensus.cc:1261] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 2 FOLLOWER]: Refusing update from remote peer 8ab12efc498b426697bb1a5531064c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:03:02.917899 22086 consensus_queue.cc:1035] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 08:03:02.942014 21753 raft_consensus.cc:1261] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 2 FOLLOWER]: Refusing update from remote peer 8ab12efc498b426697bb1a5531064c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:03:02.943069 22086 consensus_queue.cc:1035] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:03:02.951592 21902 raft_consensus.cc:1261] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166 [term 2 FOLLOWER]: Refusing update from remote peer 8ab12efc498b426697bb1a5531064c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:03:02.953311 22086 consensus_queue.cc:1035] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:03:04.421730 21704 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:04.426951 21704 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:04.427318 21704 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:06.428272 21704 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:06.431938 21902 consensus_queue.cc:237] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } }
I20240501 08:03:06.436625 21704 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:06.437265 21704 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:06.440515 21828 raft_consensus.cc:1261] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Refusing update from remote peer e282e3e8c9274b739f70534606f48166: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:03:06.441700 21753 raft_consensus.cc:1261] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Refusing update from remote peer e282e3e8c9274b739f70534606f48166: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:03:06.442992 22119 consensus_queue.cc:1035] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 08:03:06.444433 22114 consensus_queue.cc:1035] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 08:03:06.454639 22114 raft_consensus.cc:2932] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
I20240501 08:03:06.456909 21753 raft_consensus.cc:2932] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
W20240501 08:03:06.458281 21866 consensus_peers.cc:485] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 -> Peer 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4:39747): Couldn't send request to peer 568380ed8e1a4956a94c0cad49fc4a0d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63c05c8273a34209a6078c89d59bd01d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:03:06.462435 21828 raft_consensus.cc:2932] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
I20240501 08:03:06.472311 21650 catalog_manager.cc:5484] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 reported cstate change: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New cstate: current_term: 1 leader_uuid: "e282e3e8c9274b739f70534606f48166" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:06.483598 21753 consensus_queue.cc:237] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } }
I20240501 08:03:06.492568 21902 raft_consensus.cc:1261] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Refusing update from remote peer 01d3577427624fa8b733111b0dc0d119: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:03:06.498574 22130 consensus_queue.cc:1035] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 08:03:06.504369 21828 raft_consensus.cc:1261] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Refusing update from remote peer 01d3577427624fa8b733111b0dc0d119: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:03:06.506048 22097 consensus_queue.cc:1035] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240501 08:03:06.507906 21717 consensus_peers.cc:485] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 -> Peer 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4:39747): Couldn't send request to peer 568380ed8e1a4956a94c0cad49fc4a0d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44bf2a078f494657bceaca86c03dbbfc. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:03:06.511345 22130 raft_consensus.cc:2932] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
I20240501 08:03:06.513963 21902 raft_consensus.cc:2932] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
I20240501 08:03:06.517180 21828 raft_consensus.cc:2932] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
I20240501 08:03:06.526371 21902 consensus_queue.cc:237] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } attrs { replace: true } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } }
I20240501 08:03:06.526998 21651 catalog_manager.cc:5484] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166 reported cstate change: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New cstate: current_term: 1 leader_uuid: "01d3577427624fa8b733111b0dc0d119" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
W20240501 08:03:06.532675 21866 consensus_peers.cc:485] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 -> Peer 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4:39747): Couldn't send request to peer 568380ed8e1a4956a94c0cad49fc4a0d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26dc8e957a2747528356007dab573637. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:03:06.533478 21753 raft_consensus.cc:1261] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Refusing update from remote peer e282e3e8c9274b739f70534606f48166: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:03:06.533984 21828 raft_consensus.cc:1261] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Refusing update from remote peer e282e3e8c9274b739f70534606f48166: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:03:06.534845 22114 consensus_queue.cc:1035] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 08:03:06.537343 22114 consensus_queue.cc:1035] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:03:06.544060 22135 raft_consensus.cc:2932] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } attrs { replace: true } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
I20240501 08:03:06.546772 21828 raft_consensus.cc:2932] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } attrs { replace: true } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
I20240501 08:03:06.546772 21753 raft_consensus.cc:2932] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } attrs { replace: true } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
I20240501 08:03:06.555938 21649 catalog_manager.cc:5484] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f reported cstate change: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New cstate: current_term: 1 leader_uuid: "e282e3e8c9274b739f70534606f48166" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } attrs { replace: true } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
I20240501 08:03:06.560077 21752 consensus_queue.cc:237] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } }
I20240501 08:03:06.567189 21902 raft_consensus.cc:1261] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Refusing update from remote peer 01d3577427624fa8b733111b0dc0d119: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:03:06.567566 21828 raft_consensus.cc:1261] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Refusing update from remote peer 01d3577427624fa8b733111b0dc0d119: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:03:06.568737 22097 consensus_queue.cc:1035] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:03:06.569916 21717 consensus_peers.cc:485] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 -> Peer 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4:39747): Couldn't send request to peer 568380ed8e1a4956a94c0cad49fc4a0d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5710bd5ed314c1c8794e04c25b48b3e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:03:06.570401 22130 consensus_queue.cc:1035] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:03:06.576192 22097 raft_consensus.cc:2932] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
I20240501 08:03:06.579504 21828 raft_consensus.cc:2932] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
I20240501 08:03:06.584074 21902 raft_consensus.cc:2932] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } } }
I20240501 08:03:06.592315 21651 catalog_manager.cc:5484] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 reported cstate change: config changed from index -1 to 2, NON_VOTER 568380ed8e1a4956a94c0cad49fc4a0d (127.20.151.4) added. New cstate: current_term: 1 leader_uuid: "01d3577427624fa8b733111b0dc0d119" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:06.900305 22142 ts_tablet_manager.cc:902] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d: Initiating tablet copy from peer 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279)
I20240501 08:03:06.906445 22142 tablet_copy_client.cc:308] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.1:37279
I20240501 08:03:06.917949 21763 tablet_copy_service.cc:140] P 01d3577427624fa8b733111b0dc0d119: Received BeginTabletCopySession request for tablet 44bf2a078f494657bceaca86c03dbbfc from peer 568380ed8e1a4956a94c0cad49fc4a0d ({username='slave'} at 127.0.0.1:58382)
I20240501 08:03:06.918942 21763 tablet_copy_service.cc:161] P 01d3577427624fa8b733111b0dc0d119: Beginning new tablet copy session on tablet 44bf2a078f494657bceaca86c03dbbfc from peer 568380ed8e1a4956a94c0cad49fc4a0d at {username='slave'} at 127.0.0.1:58382: session id = 568380ed8e1a4956a94c0cad49fc4a0d-44bf2a078f494657bceaca86c03dbbfc
I20240501 08:03:06.932914 22148 ts_tablet_manager.cc:902] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d: Initiating tablet copy from peer e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:06.943048 21763 tablet_copy_source_session.cc:205] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:03:06.948554 22142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bf2a078f494657bceaca86c03dbbfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:06.958158 22148 tablet_copy_client.cc:308] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.3:36377
I20240501 08:03:06.967459 22142 tablet_copy_client.cc:783] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Starting download of 0 data blocks...
I20240501 08:03:06.968186 22142 tablet_copy_client.cc:647] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Starting download of 1 WAL segments...
I20240501 08:03:06.973960 22142 tablet_copy_client.cc:523] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:03:06.981113 21084 tablet_server.cc:178] TabletServer@127.20.151.1:0 shutting down...
I20240501 08:03:06.982610 22142 tablet_bootstrap.cc:492] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d: Bootstrap starting.
W20240501 08:03:06.988688 21638 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 6 similar messages]
W20240501 08:03:06.998751 21699 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:03:06.999089 21699 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 08:03:07.005595 21084 tablet_copy_service.cc:362] P 01d3577427624fa8b733111b0dc0d119: Destroying tablet copy session 568380ed8e1a4956a94c0cad49fc4a0d-44bf2a078f494657bceaca86c03dbbfc due to service shutdown
I20240501 08:03:07.006040 21084 tablet_copy_service.cc:433] P 01d3577427624fa8b733111b0dc0d119: ending tablet copy session 568380ed8e1a4956a94c0cad49fc4a0d-44bf2a078f494657bceaca86c03dbbfc on tablet 44bf2a078f494657bceaca86c03dbbfc with peer 568380ed8e1a4956a94c0cad49fc4a0d
I20240501 08:03:07.008087 22142 tablet_bootstrap.cc:492] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:03:07.009150 22142 tablet_bootstrap.cc:492] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d: Bootstrap complete.
I20240501 08:03:07.010073 22142 ts_tablet_manager.cc:1367] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.015s
W20240501 08:03:07.011281 21699 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:03:07.011725 21699 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 08:03:07.013392 22142 raft_consensus.cc:348] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } }
I20240501 08:03:07.014567 22142 raft_consensus.cc:729] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 568380ed8e1a4956a94c0cad49fc4a0d, State: Initialized, Role: LEARNER
I20240501 08:03:07.015372 22142 consensus_queue.cc:260] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } }
I20240501 08:03:07.018390 22142 ts_tablet_manager.cc:1398] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240501 08:03:07.025082 21912 tablet_copy_service.cc:140] P e282e3e8c9274b739f70534606f48166: Received BeginTabletCopySession request for tablet 26dc8e957a2747528356007dab573637 from peer 568380ed8e1a4956a94c0cad49fc4a0d ({username='slave'} at 127.0.0.1:34650)
I20240501 08:03:07.025667 21912 tablet_copy_service.cc:161] P e282e3e8c9274b739f70534606f48166: Beginning new tablet copy session on tablet 26dc8e957a2747528356007dab573637 from peer 568380ed8e1a4956a94c0cad49fc4a0d at {username='slave'} at 127.0.0.1:34650: session id = 568380ed8e1a4956a94c0cad49fc4a0d-26dc8e957a2747528356007dab573637
I20240501 08:03:07.040325 21912 tablet_copy_source_session.cc:205] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 08:03:07.060308 21794 consensus_peers.cc:485] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f -> Peer 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279): Couldn't send request to peer 01d3577427624fa8b733111b0dc0d119. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:03:07.068403 22148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26dc8e957a2747528356007dab573637. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:07.081075 22158 ts_tablet_manager.cc:902] T b5710bd5ed314c1c8794e04c25b48b3e P 568380ed8e1a4956a94c0cad49fc4a0d: Initiating tablet copy from peer 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279)
I20240501 08:03:07.109822 22155 ts_tablet_manager.cc:902] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d: Initiating tablet copy from peer e282e3e8c9274b739f70534606f48166 (127.20.151.3:36377)
I20240501 08:03:07.115459 22158 tablet_copy_client.cc:308] T b5710bd5ed314c1c8794e04c25b48b3e P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.1:37279
I20240501 08:03:07.116034 22148 tablet_copy_client.cc:783] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Starting download of 0 data blocks...
I20240501 08:03:07.116624 22148 tablet_copy_client.cc:647] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Starting download of 1 WAL segments...
I20240501 08:03:07.112234 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240501 08:03:07.119777 21868 consensus_peers.cc:485] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 -> Peer 01d3577427624fa8b733111b0dc0d119 (127.20.151.1:37279): Couldn't send request to peer 01d3577427624fa8b733111b0dc0d119. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:03:07.121006 22148 tablet_copy_client.cc:523] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:03:07.121352 22155 tablet_copy_client.cc:308] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.3:36377
I20240501 08:03:07.122072 21084 tablet_replica.cc:330] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119: stopping tablet replica
I20240501 08:03:07.122956 21084 raft_consensus.cc:2229] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:07.123385 21912 tablet_copy_service.cc:140] P e282e3e8c9274b739f70534606f48166: Received BeginTabletCopySession request for tablet 63c05c8273a34209a6078c89d59bd01d from peer 568380ed8e1a4956a94c0cad49fc4a0d ({username='slave'} at 127.0.0.1:34650)
I20240501 08:03:07.123859 21912 tablet_copy_service.cc:161] P e282e3e8c9274b739f70534606f48166: Beginning new tablet copy session on tablet 63c05c8273a34209a6078c89d59bd01d from peer 568380ed8e1a4956a94c0cad49fc4a0d at {username='slave'} at 127.0.0.1:34650: session id = 568380ed8e1a4956a94c0cad49fc4a0d-63c05c8273a34209a6078c89d59bd01d
I20240501 08:03:07.124006 21084 raft_consensus.cc:2258] T b5710bd5ed314c1c8794e04c25b48b3e P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.126993 21084 tablet_replica.cc:330] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119: stopping tablet replica
I20240501 08:03:07.127805 21084 raft_consensus.cc:2229] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.128404 21084 raft_consensus.cc:2258] T 63c05c8273a34209a6078c89d59bd01d P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.129184 22148 tablet_bootstrap.cc:492] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d: Bootstrap starting.
I20240501 08:03:07.131223 21084 tablet_replica.cc:330] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119: stopping tablet replica
I20240501 08:03:07.131712 21912 tablet_copy_source_session.cc:205] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:03:07.132105 21084 raft_consensus.cc:2229] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:07.133313 21084 raft_consensus.cc:2258] T 44bf2a078f494657bceaca86c03dbbfc P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.134619 22155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c05c8273a34209a6078c89d59bd01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:07.136384 21084 tablet_replica.cc:330] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119: stopping tablet replica
I20240501 08:03:07.137151 21084 raft_consensus.cc:2229] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.137760 21084 raft_consensus.cc:2258] T 26dc8e957a2747528356007dab573637 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.140787 21084 tablet_replica.cc:330] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119: stopping tablet replica
I20240501 08:03:07.141532 21084 raft_consensus.cc:2229] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.142115 21084 raft_consensus.cc:2258] T 234853d87e2d47edbb639e603fae75bb P 01d3577427624fa8b733111b0dc0d119 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.144444 21084 tablet_replica.cc:330] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119: stopping tablet replica
I20240501 08:03:07.145126 21084 raft_consensus.cc:2229] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:07.146066 21084 raft_consensus.cc:2258] T 0480d951fd9d40789d390a0871b46c02 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.148485 21084 tablet_replica.cc:330] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119: stopping tablet replica
I20240501 08:03:07.149164 21084 raft_consensus.cc:2229] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.149327 22155 tablet_copy_client.cc:783] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Starting download of 0 data blocks...
I20240501 08:03:07.149716 21084 raft_consensus.cc:2258] T b72589ed0fdb463daad0083d072f7e01 P 01d3577427624fa8b733111b0dc0d119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.149878 22155 tablet_copy_client.cc:647] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Starting download of 1 WAL segments...
I20240501 08:03:07.152281 21084 tablet_replica.cc:330] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119: stopping tablet replica
I20240501 08:03:07.152915 21084 raft_consensus.cc:2229] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.153427 21084 raft_consensus.cc:2258] T 99c49ce3d47344b49a81863aacc787b2 P 01d3577427624fa8b733111b0dc0d119 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.153483 22155 tablet_copy_client.cc:523] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:03:07.154569 22148 tablet_bootstrap.cc:492] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:03:07.155421 22148 tablet_bootstrap.cc:492] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d: Bootstrap complete.
I20240501 08:03:07.156100 22148 ts_tablet_manager.cc:1367] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.008s
I20240501 08:03:07.158887 22148 raft_consensus.cc:348] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } attrs { replace: true } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } }
I20240501 08:03:07.159761 22148 raft_consensus.cc:729] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 568380ed8e1a4956a94c0cad49fc4a0d, State: Initialized, Role: LEARNER
I20240501 08:03:07.160581 22148 consensus_queue.cc:260] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } attrs { replace: true } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } }
I20240501 08:03:07.161821 22155 tablet_bootstrap.cc:492] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d: Bootstrap starting.
I20240501 08:03:07.163398 22148 ts_tablet_manager.cc:1398] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:03:07.165148 21912 tablet_copy_service.cc:342] P e282e3e8c9274b739f70534606f48166: Request end of tablet copy session 568380ed8e1a4956a94c0cad49fc4a0d-26dc8e957a2747528356007dab573637 received from {username='slave'} at 127.0.0.1:34650
I20240501 08:03:07.165549 21912 tablet_copy_service.cc:433] P e282e3e8c9274b739f70534606f48166: ending tablet copy session 568380ed8e1a4956a94c0cad49fc4a0d-26dc8e957a2747528356007dab573637 on tablet 26dc8e957a2747528356007dab573637 with peer 568380ed8e1a4956a94c0cad49fc4a0d
I20240501 08:03:07.181079 22155 tablet_bootstrap.cc:492] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:03:07.182749 22155 tablet_bootstrap.cc:492] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d: Bootstrap complete.
I20240501 08:03:07.183655 22155 ts_tablet_manager.cc:1367] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.004s
I20240501 08:03:07.186278 22155 raft_consensus.cc:348] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } }
I20240501 08:03:07.187192 22155 raft_consensus.cc:729] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 568380ed8e1a4956a94c0cad49fc4a0d, State: Initialized, Role: LEARNER
W20240501 08:03:07.187892 22008 connection.cc:441] server connection from 127.0.0.1:41868 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:41868 (request call id 3) could send its response
I20240501 08:03:07.187812 22155 consensus_queue.cc:260] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "01d3577427624fa8b733111b0dc0d119" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37279 } } peers { permanent_uuid: "8ab12efc498b426697bb1a5531064c2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 44959 } attrs { replace: true } } peers { permanent_uuid: "e282e3e8c9274b739f70534606f48166" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 36377 } } peers { permanent_uuid: "568380ed8e1a4956a94c0cad49fc4a0d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 39747 } attrs { promote: true } }
I20240501 08:03:07.190644 22155 ts_tablet_manager.cc:1398] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:03:07.196573 21912 tablet_copy_service.cc:342] P e282e3e8c9274b739f70534606f48166: Request end of tablet copy session 568380ed8e1a4956a94c0cad49fc4a0d-63c05c8273a34209a6078c89d59bd01d received from {username='slave'} at 127.0.0.1:34650
I20240501 08:03:07.196970 21912 tablet_copy_service.cc:433] P e282e3e8c9274b739f70534606f48166: ending tablet copy session 568380ed8e1a4956a94c0cad49fc4a0d-63c05c8273a34209a6078c89d59bd01d on tablet 63c05c8273a34209a6078c89d59bd01d with peer 568380ed8e1a4956a94c0cad49fc4a0d
I20240501 08:03:07.209142 21084 tablet_server.cc:194] TabletServer@127.20.151.1:0 shutdown complete.
I20240501 08:03:07.231441 21084 tablet_server.cc:178] TabletServer@127.20.151.2:0 shutting down...
W20240501 08:03:07.236140 22142 tablet_copy_client.cc:1108] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.20.151.1:37279: connect: Connection refused (error 111)
I20240501 08:03:07.265656 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:03:07.266422 21084 tablet_replica.cc:330] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f: stopping tablet replica
I20240501 08:03:07.267243 21084 raft_consensus.cc:2229] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.268374 21084 raft_consensus.cc:2258] T b5710bd5ed314c1c8794e04c25b48b3e P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.272270 21084 tablet_replica.cc:330] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f: stopping tablet replica
I20240501 08:03:07.272805 21084 raft_consensus.cc:2229] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:03:07.273793 21084 raft_consensus.cc:2258] T 99c49ce3d47344b49a81863aacc787b2 P 8ab12efc498b426697bb1a5531064c2f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.275813 21084 tablet_replica.cc:330] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f: stopping tablet replica
I20240501 08:03:07.276468 21084 raft_consensus.cc:2229] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:07.277112 21084 raft_consensus.cc:2258] T b72589ed0fdb463daad0083d072f7e01 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.278954 21084 tablet_replica.cc:330] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f: stopping tablet replica
I20240501 08:03:07.279385 21084 raft_consensus.cc:2229] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.279716 21084 raft_consensus.cc:2258] T 0480d951fd9d40789d390a0871b46c02 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.281733 21084 tablet_replica.cc:330] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f: stopping tablet replica
I20240501 08:03:07.282349 21084 raft_consensus.cc:2229] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:03:07.283181 21084 raft_consensus.cc:2258] T 234853d87e2d47edbb639e603fae75bb P 8ab12efc498b426697bb1a5531064c2f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.285137 21084 tablet_replica.cc:330] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f: stopping tablet replica
I20240501 08:03:07.285573 21084 raft_consensus.cc:2229] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.285996 21084 raft_consensus.cc:2258] T 44bf2a078f494657bceaca86c03dbbfc P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.288237 21084 tablet_replica.cc:330] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f: stopping tablet replica
I20240501 08:03:07.288811 21084 raft_consensus.cc:2229] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.289217 21084 raft_consensus.cc:2258] T 26dc8e957a2747528356007dab573637 P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.291298 21084 tablet_replica.cc:330] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f: stopping tablet replica
I20240501 08:03:07.291767 21084 raft_consensus.cc:2229] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.292245 21084 raft_consensus.cc:2258] T 63c05c8273a34209a6078c89d59bd01d P 8ab12efc498b426697bb1a5531064c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.321508 21084 tablet_server.cc:194] TabletServer@127.20.151.2:0 shutdown complete.
I20240501 08:03:07.340881 21084 tablet_server.cc:178] TabletServer@127.20.151.3:0 shutting down...
W20240501 08:03:07.344791 21699 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:03:07.345872 21699 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:03:07.349316 21699 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:03:07.349619 21699 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 08:03:07.364514 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:03:07.365119 21084 tablet_replica.cc:330] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166: stopping tablet replica
I20240501 08:03:07.365628 21084 raft_consensus.cc:2229] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.366094 21084 raft_consensus.cc:2258] T b5710bd5ed314c1c8794e04c25b48b3e P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.367939 21084 tablet_replica.cc:330] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166: stopping tablet replica
I20240501 08:03:07.368399 21084 raft_consensus.cc:2229] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.368743 21084 raft_consensus.cc:2258] T 99c49ce3d47344b49a81863aacc787b2 P e282e3e8c9274b739f70534606f48166 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.370178 21084 tablet_replica.cc:330] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166: stopping tablet replica
I20240501 08:03:07.370631 21084 raft_consensus.cc:2229] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.370962 21084 raft_consensus.cc:2258] T b72589ed0fdb463daad0083d072f7e01 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.372398 21084 tablet_replica.cc:330] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166: stopping tablet replica
I20240501 08:03:07.372828 21084 raft_consensus.cc:2229] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.373221 21084 raft_consensus.cc:2258] T 44bf2a078f494657bceaca86c03dbbfc P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.375607 21084 tablet_replica.cc:330] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166: stopping tablet replica
I20240501 08:03:07.376070 21084 raft_consensus.cc:2229] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:07.376839 21084 raft_consensus.cc:2258] T 63c05c8273a34209a6078c89d59bd01d P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.378571 21084 tablet_replica.cc:330] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166: stopping tablet replica
I20240501 08:03:07.378980 21084 raft_consensus.cc:2229] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:07.379698 21084 raft_consensus.cc:2258] T 26dc8e957a2747528356007dab573637 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.381284 21084 tablet_replica.cc:330] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166: stopping tablet replica
I20240501 08:03:07.381759 21084 raft_consensus.cc:2229] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.382097 21084 raft_consensus.cc:2258] T 0480d951fd9d40789d390a0871b46c02 P e282e3e8c9274b739f70534606f48166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.383513 21084 tablet_replica.cc:330] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166: stopping tablet replica
I20240501 08:03:07.383900 21084 raft_consensus.cc:2229] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:07.384225 21084 raft_consensus.cc:2258] T 234853d87e2d47edbb639e603fae75bb P e282e3e8c9274b739f70534606f48166 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.409452 21084 tablet_server.cc:194] TabletServer@127.20.151.3:0 shutdown complete.
I20240501 08:03:07.423661 21084 tablet_server.cc:178] TabletServer@127.20.151.4:0 shutting down...
I20240501 08:03:07.444308 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:03:07.445664 21084 tablet_replica.cc:330] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d: stopping tablet replica
I20240501 08:03:07.446434 21084 raft_consensus.cc:2229] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:03:07.446835 21084 raft_consensus.cc:2258] T 63c05c8273a34209a6078c89d59bd01d P 568380ed8e1a4956a94c0cad49fc4a0d [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:03:07.448527 21084 tablet_replica.cc:330] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d: stopping tablet replica
I20240501 08:03:07.448958 21084 raft_consensus.cc:2229] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:03:07.449278 21084 raft_consensus.cc:2258] T 26dc8e957a2747528356007dab573637 P 568380ed8e1a4956a94c0cad49fc4a0d [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:03:07.450928 21084 tablet_replica.cc:330] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d: stopping tablet replica
I20240501 08:03:07.451300 21084 raft_consensus.cc:2229] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:03:07.451587 21084 raft_consensus.cc:2258] T 44bf2a078f494657bceaca86c03dbbfc P 568380ed8e1a4956a94c0cad49fc4a0d [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:03:07.473107 21084 tablet_server.cc:194] TabletServer@127.20.151.4:0 shutdown complete.
I20240501 08:03:07.482652 21084 master.cc:537] Master@127.20.151.62:34829 shutting down...
I20240501 08:03:07.498427 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:07.498859 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:07.499164 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P 56ec874134e24b8fa93635893c66c655: stopping tablet replica
I20240501 08:03:07.516219 21084 master.cc:552] Master@127.20.151.62:34829 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (9274 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240501 08:03:07.555279 21084 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.151.62:36779,127.20.151.61:33023,127.20.151.60:40815
I20240501 08:03:07.556607 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:07.562709 22172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:07.563674 22173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:07.564450 22175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:07.565304 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:07.566468 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:07.566665 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:07.566807 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550587566788 us; error 0 us; skew 500 ppm
I20240501 08:03:07.567276 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:07.569517 21084 webserver.cc:460] Webserver started at http://127.20.151.62:45975/ using document root <none> and password file <none>
I20240501 08:03:07.569958 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:07.570128 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:07.570380 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:07.571348 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-0-root/instance:
uuid: "9a90c64d4efb496491bb24c8c00f77df"
format_stamp: "Formatted at 2024-05-01 08:03:07 on dist-test-slave-pwkb"
I20240501 08:03:07.575382 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:03:07.578188 22180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:07.578941 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:03:07.579159 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-0-root
uuid: "9a90c64d4efb496491bb24c8c00f77df"
format_stamp: "Formatted at 2024-05-01 08:03:07 on dist-test-slave-pwkb"
I20240501 08:03:07.579378 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:07.614517 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:07.615491 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:07.650410 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:36779
I20240501 08:03:07.650506 22231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:36779 every 8 connection(s)
I20240501 08:03:07.653928 22232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:07.654210 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:07.659981 22234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:07.660161 22232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:03:07.661808 22235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:07.663597 22237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:07.664417 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:07.665148 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:07.665325 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:07.665459 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550587665445 us; error 0 us; skew 500 ppm
I20240501 08:03:07.665910 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:07.669867 21084 webserver.cc:460] Webserver started at http://127.20.151.61:42283/ using document root <none> and password file <none>
I20240501 08:03:07.670385 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:07.670621 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:07.670922 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:07.672158 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-1-root/instance:
uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a"
format_stamp: "Formatted at 2024-05-01 08:03:07 on dist-test-slave-pwkb"
I20240501 08:03:07.672523 22232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:03:07.677075 22232 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.151.61:33023: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33023: connect: Connection refused (error 111)
I20240501 08:03:07.677208 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240501 08:03:07.680485 22245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:07.681107 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:03:07.681341 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-1-root
uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a"
format_stamp: "Formatted at 2024-05-01 08:03:07 on dist-test-slave-pwkb"
I20240501 08:03:07.681603 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:07.700551 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:07.701566 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:07.735953 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.61:33023
I20240501 08:03:07.736037 22296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.61:33023 every 8 connection(s)
I20240501 08:03:07.738515 22232 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } attempt: 1
I20240501 08:03:07.739620 22297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:07.739722 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:07.745647 22300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:07.750159 22297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:03:07.750602 22301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:07.758136 22232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:03:07.760973 22303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:07.763094 21084 server_base.cc:1034] running on GCE node
W20240501 08:03:07.763712 22232 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.151.60:40815: Network error: Client connection negotiation failed: client connection to 127.20.151.60:40815: connect: Connection refused (error 111)
I20240501 08:03:07.764199 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:07.764427 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:07.764618 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550587764600 us; error 0 us; skew 500 ppm
I20240501 08:03:07.765281 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:07.768770 21084 webserver.cc:460] Webserver started at http://127.20.151.60:43495/ using document root <none> and password file <none>
I20240501 08:03:07.769301 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:07.769290 22297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:03:07.769632 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:07.769950 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:07.771214 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-2-root/instance:
uuid: "a7cff062f7cf4550a04ed532d516be3c"
format_stamp: "Formatted at 2024-05-01 08:03:07 on dist-test-slave-pwkb"
I20240501 08:03:07.775552 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:03:07.778311 22297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:03:07.779013 22309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:07.779924 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:03:07.780241 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-2-root
uuid: "a7cff062f7cf4550a04ed532d516be3c"
format_stamp: "Formatted at 2024-05-01 08:03:07 on dist-test-slave-pwkb"
I20240501 08:03:07.780516 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:07.790197 22232 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } attempt: 1
W20240501 08:03:07.792961 22297 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.151.60:40815: Network error: Client connection negotiation failed: client connection to 127.20.151.60:40815: connect: Connection refused (error 111)
I20240501 08:03:07.793154 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:07.794236 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240501 08:03:07.794236 22232 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.151.60:40815: Network error: Client connection negotiation failed: client connection to 127.20.151.60:40815: connect: Connection refused (error 111)
I20240501 08:03:07.815543 22297 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } attempt: 1
W20240501 08:03:07.820135 22297 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.20.151.60:40815: Network error: Client connection negotiation failed: client connection to 127.20.151.60:40815: connect: Connection refused (error 111)
I20240501 08:03:07.830025 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.60:40815
I20240501 08:03:07.830116 22361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.60:40815 every 8 connection(s)
I20240501 08:03:07.832773 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:03:07.833353 22362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:07.838511 22362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:03:07.844933 22232 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } attempt: 2
I20240501 08:03:07.848567 22362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:03:07.857964 22362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:03:07.861447 22232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df: Bootstrap starting.
I20240501 08:03:07.865547 22232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:07.869608 22232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df: No bootstrap required, opened a new log
I20240501 08:03:07.870582 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c: Bootstrap starting.
I20240501 08:03:07.871450 22232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:07.871976 22232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:07.872184 22232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a90c64d4efb496491bb24c8c00f77df, State: Initialized, Role: FOLLOWER
I20240501 08:03:07.872704 22232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:07.874604 22362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:07.874480 22367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } } }
I20240501 08:03:07.875276 22367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:03:07.875913 22232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:03:07.879029 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c: No bootstrap required, opened a new log
I20240501 08:03:07.880967 22362 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:07.881618 22362 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:07.881934 22362 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7cff062f7cf4550a04ed532d516be3c, State: Initialized, Role: FOLLOWER
I20240501 08:03:07.882464 22362 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:07.884222 22374 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } } }
I20240501 08:03:07.885025 22374 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:03:07.885722 22362 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:03:07.889081 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240501 08:03:07.894251 22380 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:03:07.894619 22380 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 08:03:07.897716 22297 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } attempt: 2
W20240501 08:03:07.900696 22391 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:03:07.900949 22391 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 08:03:07.910785 22297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a: Bootstrap starting.
I20240501 08:03:07.914597 22297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:07.918198 22297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a: No bootstrap required, opened a new log
I20240501 08:03:07.919796 22297 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:07.920251 22297 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:07.920485 22297 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6ba5e880f84d2f9a4f3d0089a70c0a, State: Initialized, Role: FOLLOWER
I20240501 08:03:07.920954 22297 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:07.922525 22393 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } } }
I20240501 08:03:07.923099 22393 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:03:07.923583 22297 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:03:07.929899 22374 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:07.930462 22374 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:07.933362 22374 leader_election.cc:288] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a90c64d4efb496491bb24c8c00f77df (127.20.151.62:36779), dc6ba5e880f84d2f9a4f3d0089a70c0a (127.20.151.61:33023)
I20240501 08:03:07.933650 22207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7cff062f7cf4550a04ed532d516be3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a90c64d4efb496491bb24c8c00f77df" is_pre_election: true
I20240501 08:03:07.933941 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7cff062f7cf4550a04ed532d516be3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" is_pre_election: true
I20240501 08:03:07.934463 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240501 08:03:07.934553 22272 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7cff062f7cf4550a04ed532d516be3c in term 0.
I20240501 08:03:07.934496 22207 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7cff062f7cf4550a04ed532d516be3c in term 0.
I20240501 08:03:07.936065 22312 leader_election.cc:302] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7cff062f7cf4550a04ed532d516be3c, dc6ba5e880f84d2f9a4f3d0089a70c0a; no voters:
I20240501 08:03:07.936365 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:03:07.936816 22374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:03:07.937120 22374 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:03:07.937366 22374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:03:07.939113 22404 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:03:07.939412 22404 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 08:03:07.942440 22374 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
W20240501 08:03:07.943789 22405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:07.945170 22374 leader_election.cc:288] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [CANDIDATE]: Term 1 election: Requested vote from peers 9a90c64d4efb496491bb24c8c00f77df (127.20.151.62:36779), dc6ba5e880f84d2f9a4f3d0089a70c0a (127.20.151.61:33023)
W20240501 08:03:07.945235 22406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:07.945230 22207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7cff062f7cf4550a04ed532d516be3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a90c64d4efb496491bb24c8c00f77df"
I20240501 08:03:07.946099 22207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:07.946169 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7cff062f7cf4550a04ed532d516be3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a"
I20240501 08:03:07.946786 22272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:03:07.947367 22408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:07.952198 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:07.952935 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 08:03:07.952972 22207 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7cff062f7cf4550a04ed532d516be3c in term 1.
W20240501 08:03:07.953166 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:07.953090 22272 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7cff062f7cf4550a04ed532d516be3c in term 1.
I20240501 08:03:07.953663 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550587953641 us; error 0 us; skew 500 ppm
I20240501 08:03:07.954170 22313 leader_election.cc:302] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a90c64d4efb496491bb24c8c00f77df, a7cff062f7cf4550a04ed532d516be3c; no voters:
I20240501 08:03:07.954504 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:07.954972 22374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:07.955951 22374 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 1 LEADER]: Becoming Leader. State: Replica: a7cff062f7cf4550a04ed532d516be3c, State: Running, Role: LEADER
I20240501 08:03:07.956771 22374 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:07.957662 21084 webserver.cc:460] Webserver started at http://127.20.151.1:37343/ using document root <none> and password file <none>
I20240501 08:03:07.958294 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:07.958534 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:07.958832 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:07.960441 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-0-root/instance:
uuid: "7be730c9495040c4bc1253843445861c"
format_stamp: "Formatted at 2024-05-01 08:03:07 on dist-test-slave-pwkb"
I20240501 08:03:07.962977 22410 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7cff062f7cf4550a04ed532d516be3c. Latest consensus state: current_term: 1 leader_uuid: "a7cff062f7cf4550a04ed532d516be3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } } }
I20240501 08:03:07.963548 22410 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [sys.catalog]: This master's current role is: LEADER
I20240501 08:03:07.966586 22414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:03:07.967497 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.006s
I20240501 08:03:07.970889 22414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:03:07.970971 22416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:07.971841 21084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:03:07.972096 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-0-root
uuid: "7be730c9495040c4bc1253843445861c"
format_stamp: "Formatted at 2024-05-01 08:03:07 on dist-test-slave-pwkb"
I20240501 08:03:07.972327 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:07.978467 22272 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 1 FOLLOWER]: Refusing update from remote peer a7cff062f7cf4550a04ed532d516be3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:03:07.979194 22207 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 1 FOLLOWER]: Refusing update from remote peer a7cff062f7cf4550a04ed532d516be3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:03:07.979676 22410 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:07.980415 22374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:07.989735 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:07.991437 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:07.990963 22367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7cff062f7cf4550a04ed532d516be3c. Latest consensus state: current_term: 1 leader_uuid: "a7cff062f7cf4550a04ed532d516be3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } } }
I20240501 08:03:07.991969 22367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:03:07.993253 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:03:07.995019 22393 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7cff062f7cf4550a04ed532d516be3c. Latest consensus state: current_term: 1 leader_uuid: "a7cff062f7cf4550a04ed532d516be3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } } }
I20240501 08:03:07.995720 22393 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:03:07.997088 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:03:07.997428 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:07.997772 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:03:07.998026 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:08.008028 22414 catalog_manager.cc:1347] Generated new cluster ID: 07a6ad646b94464c9985d38af28a66a4
I20240501 08:03:08.008373 22414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:03:08.009095 22410 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7cff062f7cf4550a04ed532d516be3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } } }
I20240501 08:03:08.009860 22410 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [sys.catalog]: This master's current role is: LEADER
I20240501 08:03:08.010090 22393 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7cff062f7cf4550a04ed532d516be3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } } }
I20240501 08:03:08.011078 22393 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:03:08.013759 22367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7cff062f7cf4550a04ed532d516be3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } } }
I20240501 08:03:08.014441 22367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:03:08.013950 22410 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7cff062f7cf4550a04ed532d516be3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } } }
I20240501 08:03:08.015137 22410 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [sys.catalog]: This master's current role is: LEADER
I20240501 08:03:08.035261 22414 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:03:08.036837 22414 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:03:08.064726 22414 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c: Generated new TSK 0
I20240501 08:03:08.065389 22414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:03:08.097157 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:45395
I20240501 08:03:08.097278 22487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:45395 every 8 connection(s)
I20240501 08:03:08.109604 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:08.119364 22495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:08.120102 22496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:08.122582 22488 heartbeater.cc:347] Connected to a master server at 127.20.151.60:40815
I20240501 08:03:08.123171 22488 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:08.124294 22488 heartbeater.cc:511] Master 127.20.151.60:40815 requested a full tablet report, sending...
I20240501 08:03:08.127566 22327 ts_manager.cc:194] Registered new tserver with Master: 7be730c9495040c4bc1253843445861c (127.20.151.1:45395)
I20240501 08:03:08.130488 22490 heartbeater.cc:347] Connected to a master server at 127.20.151.62:36779
I20240501 08:03:08.130722 22327 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40994
I20240501 08:03:08.131028 22490 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:08.132246 22490 heartbeater.cc:511] Master 127.20.151.62:36779 requested a full tablet report, sending...
I20240501 08:03:08.136312 22197 ts_manager.cc:194] Registered new tserver with Master: 7be730c9495040c4bc1253843445861c (127.20.151.1:45395)
I20240501 08:03:08.137310 22489 heartbeater.cc:347] Connected to a master server at 127.20.151.61:33023
I20240501 08:03:08.137841 22489 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:08.138725 22489 heartbeater.cc:511] Master 127.20.151.61:33023 requested a full tablet report, sending...
I20240501 08:03:08.141641 22262 ts_manager.cc:194] Registered new tserver with Master: 7be730c9495040c4bc1253843445861c (127.20.151.1:45395)
W20240501 08:03:08.142002 22498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:08.143175 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:08.143985 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:08.144233 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:08.144392 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550588144376 us; error 0 us; skew 500 ppm
I20240501 08:03:08.144891 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:08.147315 21084 webserver.cc:460] Webserver started at http://127.20.151.2:46061/ using document root <none> and password file <none>
I20240501 08:03:08.147754 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:08.147922 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:08.148170 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:08.149156 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-1-root/instance:
uuid: "972ce17080354d2b8ef800601dffbdde"
format_stamp: "Formatted at 2024-05-01 08:03:08 on dist-test-slave-pwkb"
I20240501 08:03:08.153367 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:03:08.156330 22503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:08.157039 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:03:08.157318 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-1-root
uuid: "972ce17080354d2b8ef800601dffbdde"
format_stamp: "Formatted at 2024-05-01 08:03:08 on dist-test-slave-pwkb"
I20240501 08:03:08.157577 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:08.166903 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:08.167991 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:08.169363 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:03:08.171607 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:03:08.171800 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:08.172034 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:03:08.172189 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:08.224242 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:46827
I20240501 08:03:08.224339 22565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:46827 every 8 connection(s)
I20240501 08:03:08.244680 22566 heartbeater.cc:347] Connected to a master server at 127.20.151.60:40815
I20240501 08:03:08.245225 22566 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:08.246414 22566 heartbeater.cc:511] Master 127.20.151.60:40815 requested a full tablet report, sending...
I20240501 08:03:08.249426 22327 ts_manager.cc:194] Registered new tserver with Master: 972ce17080354d2b8ef800601dffbdde (127.20.151.2:46827)
I20240501 08:03:08.251508 22327 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40996
I20240501 08:03:08.251705 22567 heartbeater.cc:347] Connected to a master server at 127.20.151.61:33023
I20240501 08:03:08.252110 22567 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:08.252583 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:03:08.252894 22567 heartbeater.cc:511] Master 127.20.151.61:33023 requested a full tablet report, sending...
I20240501 08:03:08.255671 22262 ts_manager.cc:194] Registered new tserver with Master: 972ce17080354d2b8ef800601dffbdde (127.20.151.2:46827)
W20240501 08:03:08.260536 22573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:08.262113 22574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:08.263758 22568 heartbeater.cc:347] Connected to a master server at 127.20.151.62:36779
I20240501 08:03:08.264277 22568 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:08.265095 22568 heartbeater.cc:511] Master 127.20.151.62:36779 requested a full tablet report, sending...
I20240501 08:03:08.267652 22197 ts_manager.cc:194] Registered new tserver with Master: 972ce17080354d2b8ef800601dffbdde (127.20.151.2:46827)
W20240501 08:03:08.268849 22576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:08.269513 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:08.270272 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:08.270452 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:08.270592 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550588270581 us; error 0 us; skew 500 ppm
I20240501 08:03:08.271044 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:08.273361 21084 webserver.cc:460] Webserver started at http://127.20.151.3:36065/ using document root <none> and password file <none>
I20240501 08:03:08.273788 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:08.273950 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:08.274185 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:08.275208 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-2-root/instance:
uuid: "a8c6f41a3a90449f89bb0d22832fc71c"
format_stamp: "Formatted at 2024-05-01 08:03:08 on dist-test-slave-pwkb"
I20240501 08:03:08.279201 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:03:08.282459 22581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:08.283327 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:03:08.283579 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-2-root
uuid: "a8c6f41a3a90449f89bb0d22832fc71c"
format_stamp: "Formatted at 2024-05-01 08:03:08 on dist-test-slave-pwkb"
I20240501 08:03:08.283812 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550573434241-21084-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:08.294735 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:08.295779 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:08.296846 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:03:08.298885 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:03:08.299070 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:08.299295 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:03:08.299428 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:08.335440 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:38135
I20240501 08:03:08.335526 22643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:38135 every 8 connection(s)
I20240501 08:03:08.356328 22644 heartbeater.cc:347] Connected to a master server at 127.20.151.60:40815
I20240501 08:03:08.356930 22644 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:08.358045 22644 heartbeater.cc:511] Master 127.20.151.60:40815 requested a full tablet report, sending...
I20240501 08:03:08.360858 22327 ts_manager.cc:194] Registered new tserver with Master: a8c6f41a3a90449f89bb0d22832fc71c (127.20.151.3:38135)
I20240501 08:03:08.363289 22327 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41000
I20240501 08:03:08.365159 22645 heartbeater.cc:347] Connected to a master server at 127.20.151.61:33023
I20240501 08:03:08.365759 22645 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:08.367998 22645 heartbeater.cc:511] Master 127.20.151.61:33023 requested a full tablet report, sending...
I20240501 08:03:08.370489 22262 ts_manager.cc:194] Registered new tserver with Master: a8c6f41a3a90449f89bb0d22832fc71c (127.20.151.3:38135)
I20240501 08:03:08.371037 22646 heartbeater.cc:347] Connected to a master server at 127.20.151.62:36779
I20240501 08:03:08.371569 22646 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:08.372121 22646 heartbeater.cc:511] Master 127.20.151.62:36779 requested a full tablet report, sending...
I20240501 08:03:08.374111 22197 ts_manager.cc:194] Registered new tserver with Master: a8c6f41a3a90449f89bb0d22832fc71c (127.20.151.3:38135)
I20240501 08:03:08.374248 21084 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006838226s
I20240501 08:03:08.898236 22380 catalog_manager.cc:1259] Loaded cluster ID: 07a6ad646b94464c9985d38af28a66a4
I20240501 08:03:08.898520 22380 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df: loading cluster ID for follower catalog manager: success
I20240501 08:03:08.902690 22380 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df: acquiring CA information for follower catalog manager: success
I20240501 08:03:08.905615 22380 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 08:03:08.942816 22404 catalog_manager.cc:1259] Loaded cluster ID: 07a6ad646b94464c9985d38af28a66a4
I20240501 08:03:08.943051 22404 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a: loading cluster ID for follower catalog manager: success
I20240501 08:03:08.947037 22404 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a: acquiring CA information for follower catalog manager: success
I20240501 08:03:08.949906 22404 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 08:03:09.134572 22488 heartbeater.cc:503] Master 127.20.151.60:40815 was elected leader, sending a full tablet report...
I20240501 08:03:09.254706 22566 heartbeater.cc:503] Master 127.20.151.60:40815 was elected leader, sending a full tablet report...
I20240501 08:03:09.366755 22644 heartbeater.cc:503] Master 127.20.151.60:40815 was elected leader, sending a full tablet report...
I20240501 08:03:09.897712 22390 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:09.937496 21084 test_util.cc:268] Using random seed: -354625726
I20240501 08:03:09.973135 22327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53384:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 08:03:10.034132 22531 tablet_service.cc:1467] Processing CreateTablet for tablet feca3879631a4f4e88fce2e634556fa8 (DEFAULT_TABLE table=test-workload [id=bb72d0d81103492b9ff524558ddb6bb7]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 08:03:10.035467 22531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feca3879631a4f4e88fce2e634556fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:10.052340 22453 tablet_service.cc:1467] Processing CreateTablet for tablet c965879e464c46ccbff85d0afa9df165 (DEFAULT_TABLE table=test-workload [id=bb72d0d81103492b9ff524558ddb6bb7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 08:03:10.053717 22453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c965879e464c46ccbff85d0afa9df165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:10.054308 22678 tablet_bootstrap.cc:492] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde: Bootstrap starting.
I20240501 08:03:10.059834 22678 tablet_bootstrap.cc:654] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:10.060606 22609 tablet_service.cc:1467] Processing CreateTablet for tablet 62c9610b202644c7ad254adfba05155a (DEFAULT_TABLE table=test-workload [id=bb72d0d81103492b9ff524558ddb6bb7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 08:03:10.061887 22609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c9610b202644c7ad254adfba05155a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:10.064929 22678 tablet_bootstrap.cc:492] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde: No bootstrap required, opened a new log
I20240501 08:03:10.065284 22678 ts_tablet_manager.cc:1367] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240501 08:03:10.067288 22678 raft_consensus.cc:348] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972ce17080354d2b8ef800601dffbdde" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46827 } }
I20240501 08:03:10.067664 22680 tablet_bootstrap.cc:492] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c: Bootstrap starting.
I20240501 08:03:10.068017 22678 raft_consensus.cc:374] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:10.068369 22678 raft_consensus.cc:729] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972ce17080354d2b8ef800601dffbdde, State: Initialized, Role: FOLLOWER
I20240501 08:03:10.069036 22678 consensus_queue.cc:260] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972ce17080354d2b8ef800601dffbdde" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46827 } }
I20240501 08:03:10.069669 22678 raft_consensus.cc:388] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:03:10.069988 22678 raft_consensus.cc:482] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:03:10.070351 22678 raft_consensus.cc:3037] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:10.073751 22680 tablet_bootstrap.cc:654] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:10.076861 22681 tablet_bootstrap.cc:492] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c: Bootstrap starting.
I20240501 08:03:10.077183 22678 raft_consensus.cc:504] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972ce17080354d2b8ef800601dffbdde" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46827 } }
I20240501 08:03:10.078086 22678 leader_election.cc:302] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 972ce17080354d2b8ef800601dffbdde; no voters:
I20240501 08:03:10.079420 22678 leader_election.cc:288] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:03:10.079829 22683 raft_consensus.cc:2781] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:10.080104 22680 tablet_bootstrap.cc:492] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c: No bootstrap required, opened a new log
I20240501 08:03:10.080787 22680 ts_tablet_manager.cc:1367] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240501 08:03:10.084471 22681 tablet_bootstrap.cc:654] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:10.084551 22680 raft_consensus.cc:348] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be730c9495040c4bc1253843445861c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45395 } }
I20240501 08:03:10.085664 22680 raft_consensus.cc:374] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:10.086120 22680 raft_consensus.cc:729] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be730c9495040c4bc1253843445861c, State: Initialized, Role: FOLLOWER
I20240501 08:03:10.087038 22680 consensus_queue.cc:260] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be730c9495040c4bc1253843445861c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45395 } }
I20240501 08:03:10.087901 22680 raft_consensus.cc:388] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:03:10.088251 22680 raft_consensus.cc:482] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:03:10.088631 22680 raft_consensus.cc:3037] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:10.088795 22678 ts_tablet_manager.cc:1398] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde: Time spent starting tablet: real 0.023s user 0.011s sys 0.011s
I20240501 08:03:10.089972 22683 raft_consensus.cc:686] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [term 1 LEADER]: Becoming Leader. State: Replica: 972ce17080354d2b8ef800601dffbdde, State: Running, Role: LEADER
I20240501 08:03:10.090746 22683 consensus_queue.cc:237] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972ce17080354d2b8ef800601dffbdde" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46827 } }
I20240501 08:03:10.094375 22680 raft_consensus.cc:504] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be730c9495040c4bc1253843445861c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45395 } }
I20240501 08:03:10.095156 22680 leader_election.cc:302] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7be730c9495040c4bc1253843445861c; no voters:
I20240501 08:03:10.096637 22681 tablet_bootstrap.cc:492] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c: No bootstrap required, opened a new log
I20240501 08:03:10.096580 22680 leader_election.cc:288] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:03:10.097002 22688 raft_consensus.cc:2781] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:10.097813 22681 ts_tablet_manager.cc:1367] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.009s
I20240501 08:03:10.100821 22681 raft_consensus.cc:348] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c6f41a3a90449f89bb0d22832fc71c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38135 } }
I20240501 08:03:10.101459 22327 catalog_manager.cc:5484] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde reported cstate change: term changed from 0 to 1, leader changed from <none> to 972ce17080354d2b8ef800601dffbdde (127.20.151.2). New cstate: current_term: 1 leader_uuid: "972ce17080354d2b8ef800601dffbdde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972ce17080354d2b8ef800601dffbdde" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46827 } health_report { overall_health: HEALTHY } } }
I20240501 08:03:10.102167 22681 raft_consensus.cc:374] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:10.102913 22681 raft_consensus.cc:729] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c6f41a3a90449f89bb0d22832fc71c, State: Initialized, Role: FOLLOWER
I20240501 08:03:10.111940 22688 raft_consensus.cc:686] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [term 1 LEADER]: Becoming Leader. State: Replica: 7be730c9495040c4bc1253843445861c, State: Running, Role: LEADER
I20240501 08:03:10.112776 22681 consensus_queue.cc:260] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c6f41a3a90449f89bb0d22832fc71c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38135 } }
I20240501 08:03:10.113175 22688 consensus_queue.cc:237] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be730c9495040c4bc1253843445861c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45395 } }
I20240501 08:03:10.113587 22681 raft_consensus.cc:388] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:03:10.114091 22681 raft_consensus.cc:482] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:03:10.114480 22681 raft_consensus.cc:3037] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:10.120752 22681 raft_consensus.cc:504] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c6f41a3a90449f89bb0d22832fc71c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38135 } }
I20240501 08:03:10.121536 22681 leader_election.cc:302] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8c6f41a3a90449f89bb0d22832fc71c; no voters:
I20240501 08:03:10.126982 22326 catalog_manager.cc:5484] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7be730c9495040c4bc1253843445861c (127.20.151.1). New cstate: current_term: 1 leader_uuid: "7be730c9495040c4bc1253843445861c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be730c9495040c4bc1253843445861c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45395 } health_report { overall_health: HEALTHY } } }
I20240501 08:03:10.127835 22680 ts_tablet_manager.cc:1398] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c: Time spent starting tablet: real 0.047s user 0.014s sys 0.015s
I20240501 08:03:10.140218 22681 leader_election.cc:288] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:03:10.141289 22693 raft_consensus.cc:2781] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:10.161437 22681 ts_tablet_manager.cc:1398] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c: Time spent starting tablet: real 0.063s user 0.025s sys 0.010s
I20240501 08:03:10.166294 22693 raft_consensus.cc:686] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [term 1 LEADER]: Becoming Leader. State: Replica: a8c6f41a3a90449f89bb0d22832fc71c, State: Running, Role: LEADER
I20240501 08:03:10.166930 22693 consensus_queue.cc:237] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c6f41a3a90449f89bb0d22832fc71c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38135 } }
I20240501 08:03:10.172832 22327 catalog_manager.cc:5484] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c reported cstate change: term changed from 0 to 1, leader changed from <none> to a8c6f41a3a90449f89bb0d22832fc71c (127.20.151.3). New cstate: current_term: 1 leader_uuid: "a8c6f41a3a90449f89bb0d22832fc71c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c6f41a3a90449f89bb0d22832fc71c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38135 } health_report { overall_health: HEALTHY } } }
I20240501 08:03:10.192409 21084 master.cc:537] Master@127.20.151.60:40815 shutting down...
I20240501 08:03:10.207075 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:10.207914 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:10.208344 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P a7cff062f7cf4550a04ed532d516be3c: stopping tablet replica
I20240501 08:03:10.227836 21084 master.cc:552] Master@127.20.151.60:40815 shutdown complete.
W20240501 08:03:11.143266 22566 heartbeater.cc:644] Failed to heartbeat to 127.20.151.60:40815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.151.60:40815: connect: Connection refused (error 111)
I20240501 08:03:11.686195 22703 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a7cff062f7cf4550a04ed532d516be3c)
I20240501 08:03:11.686751 22704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a7cff062f7cf4550a04ed532d516be3c)
I20240501 08:03:11.686686 22703 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:11.687184 22704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:11.688300 22703 leader_election.cc:288] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a90c64d4efb496491bb24c8c00f77df (127.20.151.62:36779), a7cff062f7cf4550a04ed532d516be3c (127.20.151.60:40815)
I20240501 08:03:11.688926 22704 leader_election.cc:288] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc6ba5e880f84d2f9a4f3d0089a70c0a (127.20.151.61:33023), a7cff062f7cf4550a04ed532d516be3c (127.20.151.60:40815)
I20240501 08:03:11.689798 22206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9a90c64d4efb496491bb24c8c00f77df" is_pre_election: true
I20240501 08:03:11.690678 22206 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc6ba5e880f84d2f9a4f3d0089a70c0a in term 1.
I20240501 08:03:11.690794 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a90c64d4efb496491bb24c8c00f77df" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" is_pre_election: true
I20240501 08:03:11.691493 22271 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a90c64d4efb496491bb24c8c00f77df in term 1.
I20240501 08:03:11.692888 22182 leader_election.cc:302] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a90c64d4efb496491bb24c8c00f77df, dc6ba5e880f84d2f9a4f3d0089a70c0a; no voters:
I20240501 08:03:11.692906 22248 leader_election.cc:302] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a90c64d4efb496491bb24c8c00f77df, dc6ba5e880f84d2f9a4f3d0089a70c0a; no voters:
I20240501 08:03:11.693917 22704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:03:11.694260 22704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 1 FOLLOWER]: Starting leader election (detected failure of leader a7cff062f7cf4550a04ed532d516be3c)
I20240501 08:03:11.694568 22703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:03:11.694629 22704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:03:11.694983 22703 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 1 FOLLOWER]: Starting leader election (detected failure of leader a7cff062f7cf4550a04ed532d516be3c)
I20240501 08:03:11.695353 22703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:03:11.696355 22249 leader_election.cc:334] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a7cff062f7cf4550a04ed532d516be3c (127.20.151.60:40815): Network error: Client connection negotiation failed: client connection to 127.20.151.60:40815: connect: Connection refused (error 111)
I20240501 08:03:11.701306 22703 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:11.701306 22704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:11.703740 22704 leader_election.cc:288] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [CANDIDATE]: Term 2 election: Requested vote from peers dc6ba5e880f84d2f9a4f3d0089a70c0a (127.20.151.61:33023), a7cff062f7cf4550a04ed532d516be3c (127.20.151.60:40815)
I20240501 08:03:11.704067 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a90c64d4efb496491bb24c8c00f77df" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a"
W20240501 08:03:11.704448 22184 leader_election.cc:334] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a7cff062f7cf4550a04ed532d516be3c (127.20.151.60:40815): Network error: Client connection negotiation failed: client connection to 127.20.151.60:40815: connect: Connection refused (error 111)
I20240501 08:03:11.704821 22271 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a90c64d4efb496491bb24c8c00f77df in current term 2: Already voted for candidate dc6ba5e880f84d2f9a4f3d0089a70c0a in this term.
I20240501 08:03:11.705546 22703 leader_election.cc:288] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [CANDIDATE]: Term 2 election: Requested vote from peers 9a90c64d4efb496491bb24c8c00f77df (127.20.151.62:36779), a7cff062f7cf4550a04ed532d516be3c (127.20.151.60:40815)
I20240501 08:03:11.706444 22206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9a90c64d4efb496491bb24c8c00f77df"
I20240501 08:03:11.707230 22206 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dc6ba5e880f84d2f9a4f3d0089a70c0a in current term 2: Already voted for candidate 9a90c64d4efb496491bb24c8c00f77df in this term.
W20240501 08:03:11.707418 22249 leader_election.cc:334] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a7cff062f7cf4550a04ed532d516be3c (127.20.151.60:40815): Network error: Client connection negotiation failed: client connection to 127.20.151.60:40815: connect: Connection refused (error 111)
W20240501 08:03:11.707748 22184 leader_election.cc:334] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a7cff062f7cf4550a04ed532d516be3c (127.20.151.60:40815): Network error: Client connection negotiation failed: client connection to 127.20.151.60:40815: connect: Connection refused (error 111)
I20240501 08:03:11.708112 22184 leader_election.cc:302] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a90c64d4efb496491bb24c8c00f77df; no voters: a7cff062f7cf4550a04ed532d516be3c, dc6ba5e880f84d2f9a4f3d0089a70c0a
I20240501 08:03:11.708659 22704 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 08:03:11.709151 22248 leader_election.cc:302] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc6ba5e880f84d2f9a4f3d0089a70c0a; no voters: 9a90c64d4efb496491bb24c8c00f77df, a7cff062f7cf4550a04ed532d516be3c
I20240501 08:03:11.709731 22703 raft_consensus.cc:2726] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 08:03:13.633421 22715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:13.633893 22715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:13.635660 22715 leader_election.cc:288] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dc6ba5e880f84d2f9a4f3d0089a70c0a (127.20.151.61:33023), a7cff062f7cf4550a04ed532d516be3c (127.20.151.60:40815)
I20240501 08:03:13.636404 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a90c64d4efb496491bb24c8c00f77df" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" is_pre_election: true
I20240501 08:03:13.636914 22271 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a90c64d4efb496491bb24c8c00f77df in term 2.
I20240501 08:03:13.637914 22182 leader_election.cc:302] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a90c64d4efb496491bb24c8c00f77df, dc6ba5e880f84d2f9a4f3d0089a70c0a; no voters:
W20240501 08:03:13.638121 22184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.60:40815: connect: Connection refused (error 111) [suppressed 15 similar messages]
I20240501 08:03:13.638643 22715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 08:03:13.639039 22715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:03:13.639425 22715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 2 FOLLOWER]: Advancing to term 3
W20240501 08:03:13.641147 22184 leader_election.cc:334] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a7cff062f7cf4550a04ed532d516be3c (127.20.151.60:40815): Network error: Client connection negotiation failed: client connection to 127.20.151.60:40815: connect: Connection refused (error 111)
I20240501 08:03:13.644584 22715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:13.645931 22715 leader_election.cc:288] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [CANDIDATE]: Term 3 election: Requested vote from peers dc6ba5e880f84d2f9a4f3d0089a70c0a (127.20.151.61:33023), a7cff062f7cf4550a04ed532d516be3c (127.20.151.60:40815)
I20240501 08:03:13.646556 22271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a90c64d4efb496491bb24c8c00f77df" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a"
I20240501 08:03:13.647023 22271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 2 FOLLOWER]: Advancing to term 3
W20240501 08:03:13.649351 22184 leader_election.cc:334] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a7cff062f7cf4550a04ed532d516be3c (127.20.151.60:40815): Network error: Client connection negotiation failed: client connection to 127.20.151.60:40815: connect: Connection refused (error 111)
I20240501 08:03:13.653287 22271 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a90c64d4efb496491bb24c8c00f77df in term 3.
I20240501 08:03:13.682148 22182 leader_election.cc:302] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a90c64d4efb496491bb24c8c00f77df, dc6ba5e880f84d2f9a4f3d0089a70c0a; no voters: a7cff062f7cf4550a04ed532d516be3c
I20240501 08:03:13.686231 22715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:03:13.687287 22715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 3 LEADER]: Becoming Leader. State: Replica: 9a90c64d4efb496491bb24c8c00f77df, State: Running, Role: LEADER
I20240501 08:03:13.688057 22715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } }
I20240501 08:03:13.691777 22719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a90c64d4efb496491bb24c8c00f77df. Latest consensus state: current_term: 3 leader_uuid: "9a90c64d4efb496491bb24c8c00f77df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a90c64d4efb496491bb24c8c00f77df" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36779 } } peers { permanent_uuid: "dc6ba5e880f84d2f9a4f3d0089a70c0a" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33023 } } peers { permanent_uuid: "a7cff062f7cf4550a04ed532d516be3c" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 40815 } } }
I20240501 08:03:13.692425 22719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [sys.catalog]: This master's current role is: LEADER
I20240501 08:03:13.693951 22721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:03:13.699772 22721 catalog_manager.cc:669] Loaded metadata for table test-workload [id=bb72d0d81103492b9ff524558ddb6bb7]
I20240501 08:03:13.705529 22721 tablet_loader.cc:96] loaded metadata for tablet 62c9610b202644c7ad254adfba05155a (table test-workload [id=bb72d0d81103492b9ff524558ddb6bb7])
I20240501 08:03:13.706619 22721 tablet_loader.cc:96] loaded metadata for tablet c965879e464c46ccbff85d0afa9df165 (table test-workload [id=bb72d0d81103492b9ff524558ddb6bb7])
I20240501 08:03:13.707507 22721 tablet_loader.cc:96] loaded metadata for tablet feca3879631a4f4e88fce2e634556fa8 (table test-workload [id=bb72d0d81103492b9ff524558ddb6bb7])
I20240501 08:03:13.708351 22721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:03:13.711236 22721 catalog_manager.cc:1259] Loaded cluster ID: 07a6ad646b94464c9985d38af28a66a4
I20240501 08:03:13.711441 22721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:03:13.715598 22721 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:03:13.718534 22721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df: Loaded TSK: 0
I20240501 08:03:13.719374 22721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:03:13.890301 22379 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:13.890638 22379 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:13.918572 21084 tablet_server.cc:178] TabletServer@127.20.151.1:0 shutting down...
I20240501 08:03:13.940861 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:03:13.941519 21084 tablet_replica.cc:330] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c: stopping tablet replica
I20240501 08:03:13.942124 21084 raft_consensus.cc:2229] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:13.942660 21084 raft_consensus.cc:2258] T c965879e464c46ccbff85d0afa9df165 P 7be730c9495040c4bc1253843445861c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:13.961390 21084 tablet_server.cc:194] TabletServer@127.20.151.1:0 shutdown complete.
I20240501 08:03:13.971803 21084 tablet_server.cc:178] TabletServer@127.20.151.2:0 shutting down...
I20240501 08:03:13.996675 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:03:13.997354 21084 tablet_replica.cc:330] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde: stopping tablet replica
I20240501 08:03:13.997941 21084 raft_consensus.cc:2229] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:13.998451 21084 raft_consensus.cc:2258] T feca3879631a4f4e88fce2e634556fa8 P 972ce17080354d2b8ef800601dffbdde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:14.017011 21084 tablet_server.cc:194] TabletServer@127.20.151.2:0 shutdown complete.
I20240501 08:03:14.026575 21084 tablet_server.cc:178] TabletServer@127.20.151.3:0 shutting down...
I20240501 08:03:14.049130 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:03:14.049875 21084 tablet_replica.cc:330] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c: stopping tablet replica
I20240501 08:03:14.050359 21084 raft_consensus.cc:2229] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:14.050880 21084 raft_consensus.cc:2258] T 62c9610b202644c7ad254adfba05155a P a8c6f41a3a90449f89bb0d22832fc71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:14.068807 21084 tablet_server.cc:194] TabletServer@127.20.151.3:0 shutdown complete.
I20240501 08:03:14.077734 21084 master.cc:537] Master@127.20.151.62:36779 shutting down...
I20240501 08:03:14.098444 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 3 LEADER]: Raft consensus shutting down.
I20240501 08:03:14.099126 21084 pending_rounds.cc:62] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df: Trying to abort 1 pending ops.
I20240501 08:03:14.099339 21084 pending_rounds.cc:68] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df: Aborting op as it isn't in flight: id { term: 3 index: 10 } timestamp: 7022799231753838592 op_type: NO_OP noop_request { }
I20240501 08:03:14.099786 21084 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 08:03:14.100059 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:14.100394 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9a90c64d4efb496491bb24c8c00f77df: stopping tablet replica
I20240501 08:03:14.119288 21084 master.cc:552] Master@127.20.151.62:36779 shutdown complete.
I20240501 08:03:14.132647 21084 master.cc:537] Master@127.20.151.61:33023 shutting down...
I20240501 08:03:14.152969 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:14.153532 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:14.153854 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P dc6ba5e880f84d2f9a4f3d0089a70c0a: stopping tablet replica
I20240501 08:03:14.171339 21084 master.cc:552] Master@127.20.151.61:33023 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6647 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240501 08:03:14.202932 21084 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.151.62:34821
I20240501 08:03:14.203938 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:14.210345 22722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:14.211246 22723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:14.212173 22725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:14.212910 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:14.213608 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:14.213867 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:14.214047 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550594214023 us; error 0 us; skew 500 ppm
I20240501 08:03:14.214691 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:14.221168 21084 webserver.cc:460] Webserver started at http://127.20.151.62:33609/ using document root <none> and password file <none>
I20240501 08:03:14.221565 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:14.221746 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:14.221984 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:14.223157 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/master-0-root/instance:
uuid: "e91c8826265b48fa8e9c49fe52eca850"
format_stamp: "Formatted at 2024-05-01 08:03:14 on dist-test-slave-pwkb"
I20240501 08:03:14.227270 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:03:14.230324 22730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:14.231026 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:03:14.231271 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/master-0-root
uuid: "e91c8826265b48fa8e9c49fe52eca850"
format_stamp: "Formatted at 2024-05-01 08:03:14 on dist-test-slave-pwkb"
I20240501 08:03:14.231527 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:14.255262 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:14.256441 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:14.292181 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:34821
I20240501 08:03:14.292268 22781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:34821 every 8 connection(s)
I20240501 08:03:14.296029 22782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:14.306151 22782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850: Bootstrap starting.
I20240501 08:03:14.310144 22782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:14.314085 22782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850: No bootstrap required, opened a new log
I20240501 08:03:14.315927 22782 raft_consensus.cc:348] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91c8826265b48fa8e9c49fe52eca850" member_type: VOTER }
I20240501 08:03:14.316355 22782 raft_consensus.cc:374] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:14.316640 22782 raft_consensus.cc:729] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91c8826265b48fa8e9c49fe52eca850, State: Initialized, Role: FOLLOWER
I20240501 08:03:14.317296 22782 consensus_queue.cc:260] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91c8826265b48fa8e9c49fe52eca850" member_type: VOTER }
I20240501 08:03:14.317772 22782 raft_consensus.cc:388] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:03:14.318044 22782 raft_consensus.cc:482] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:03:14.318264 22782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:14.321991 22782 raft_consensus.cc:504] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91c8826265b48fa8e9c49fe52eca850" member_type: VOTER }
I20240501 08:03:14.322422 22782 leader_election.cc:302] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e91c8826265b48fa8e9c49fe52eca850; no voters:
I20240501 08:03:14.323415 22782 leader_election.cc:288] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:03:14.323762 22785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:14.325019 22785 raft_consensus.cc:686] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [term 1 LEADER]: Becoming Leader. State: Replica: e91c8826265b48fa8e9c49fe52eca850, State: Running, Role: LEADER
I20240501 08:03:14.325525 22785 consensus_queue.cc:237] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91c8826265b48fa8e9c49fe52eca850" member_type: VOTER }
I20240501 08:03:14.326179 22782 sys_catalog.cc:564] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:03:14.328220 22787 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e91c8826265b48fa8e9c49fe52eca850. Latest consensus state: current_term: 1 leader_uuid: "e91c8826265b48fa8e9c49fe52eca850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91c8826265b48fa8e9c49fe52eca850" member_type: VOTER } }
I20240501 08:03:14.328202 22786 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e91c8826265b48fa8e9c49fe52eca850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91c8826265b48fa8e9c49fe52eca850" member_type: VOTER } }
I20240501 08:03:14.328871 22787 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [sys.catalog]: This master's current role is: LEADER
I20240501 08:03:14.328969 22786 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [sys.catalog]: This master's current role is: LEADER
I20240501 08:03:14.337291 22790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:03:14.341710 22790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:03:14.342638 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:03:14.349609 22790 catalog_manager.cc:1347] Generated new cluster ID: 45579659f628466896701fcac280fa99
I20240501 08:03:14.349858 22790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:03:14.366468 22790 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:03:14.367545 22790 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:03:14.378315 22790 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850: Generated new TSK 0
I20240501 08:03:14.378790 22790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:03:14.409093 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:14.415587 22803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:14.416443 22804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:14.417719 21084 server_base.cc:1034] running on GCE node
W20240501 08:03:14.418557 22806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:14.419282 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:14.419490 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:14.419644 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550594419625 us; error 0 us; skew 500 ppm
I20240501 08:03:14.420152 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:14.422454 21084 webserver.cc:460] Webserver started at http://127.20.151.1:44759/ using document root <none> and password file <none>
I20240501 08:03:14.422914 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:14.423085 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:14.423326 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:14.424347 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-0-root/instance:
uuid: "1cb6f91c8b73428babc8d9ef41f4a42b"
format_stamp: "Formatted at 2024-05-01 08:03:14 on dist-test-slave-pwkb"
I20240501 08:03:14.428776 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:03:14.431805 22811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:14.432699 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:03:14.432991 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-0-root
uuid: "1cb6f91c8b73428babc8d9ef41f4a42b"
format_stamp: "Formatted at 2024-05-01 08:03:14 on dist-test-slave-pwkb"
I20240501 08:03:14.433334 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:14.447252 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:14.448370 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:14.449878 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:03:14.452180 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:03:14.452368 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:14.452587 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:03:14.452759 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:14.490195 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:36009
I20240501 08:03:14.490283 22873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:36009 every 8 connection(s)
I20240501 08:03:14.494706 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:14.503646 22878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:14.504555 22879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:14.505592 22881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:14.506579 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:14.507612 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:14.507794 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:14.507963 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550594507945 us; error 0 us; skew 500 ppm
I20240501 08:03:14.508237 22874 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34821
I20240501 08:03:14.508550 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:14.508587 22874 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:14.509765 22874 heartbeater.cc:511] Master 127.20.151.62:34821 requested a full tablet report, sending...
I20240501 08:03:14.511695 21084 webserver.cc:460] Webserver started at http://127.20.151.2:43625/ using document root <none> and password file <none>
I20240501 08:03:14.511958 22747 ts_manager.cc:194] Registered new tserver with Master: 1cb6f91c8b73428babc8d9ef41f4a42b (127.20.151.1:36009)
I20240501 08:03:14.512363 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:14.512627 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:14.512959 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:14.514286 22747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57680
I20240501 08:03:14.514367 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-1-root/instance:
uuid: "4877889d29a3461d9c9b25081fa3568e"
format_stamp: "Formatted at 2024-05-01 08:03:14 on dist-test-slave-pwkb"
I20240501 08:03:14.518980 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 08:03:14.522112 22886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:14.522830 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240501 08:03:14.523053 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-1-root
uuid: "4877889d29a3461d9c9b25081fa3568e"
format_stamp: "Formatted at 2024-05-01 08:03:14 on dist-test-slave-pwkb"
I20240501 08:03:14.523277 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:14.551785 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:14.552814 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:14.554131 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:03:14.556157 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:03:14.556342 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:14.556586 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:03:14.556730 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:14.592998 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:33869
I20240501 08:03:14.593082 22948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:33869 every 8 connection(s)
I20240501 08:03:14.597477 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:14.604732 22952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:14.605633 22953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:14.608042 22955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:14.608249 22949 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34821
I20240501 08:03:14.608672 22949 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:14.608999 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:14.609380 22949 heartbeater.cc:511] Master 127.20.151.62:34821 requested a full tablet report, sending...
I20240501 08:03:14.609745 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:14.609961 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:14.610119 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550594610102 us; error 0 us; skew 500 ppm
I20240501 08:03:14.610833 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:14.611239 22747 ts_manager.cc:194] Registered new tserver with Master: 4877889d29a3461d9c9b25081fa3568e (127.20.151.2:33869)
I20240501 08:03:14.612632 22747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57694
I20240501 08:03:14.613670 21084 webserver.cc:460] Webserver started at http://127.20.151.3:43249/ using document root <none> and password file <none>
I20240501 08:03:14.614171 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:14.614332 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:14.614560 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:14.615626 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-2-root/instance:
uuid: "d995ca92c4d247118c74bde8dfda226c"
format_stamp: "Formatted at 2024-05-01 08:03:14 on dist-test-slave-pwkb"
I20240501 08:03:14.619663 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 08:03:14.622689 22960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:14.623432 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:03:14.623688 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-2-root
uuid: "d995ca92c4d247118c74bde8dfda226c"
format_stamp: "Formatted at 2024-05-01 08:03:14 on dist-test-slave-pwkb"
I20240501 08:03:14.623934 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:14.636646 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:14.637632 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:14.638978 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:03:14.640975 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:03:14.641144 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:14.641371 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:03:14.641508 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:14.677927 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:46073
I20240501 08:03:14.678026 23022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:46073 every 8 connection(s)
I20240501 08:03:14.690032 23023 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34821
I20240501 08:03:14.690336 23023 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:14.690994 23023 heartbeater.cc:511] Master 127.20.151.62:34821 requested a full tablet report, sending...
I20240501 08:03:14.692618 22747 ts_manager.cc:194] Registered new tserver with Master: d995ca92c4d247118c74bde8dfda226c (127.20.151.3:46073)
I20240501 08:03:14.692965 21084 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012005938s
I20240501 08:03:14.693890 22747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57702
I20240501 08:03:15.516767 22874 heartbeater.cc:503] Master 127.20.151.62:34821 was elected leader, sending a full tablet report...
I20240501 08:03:15.614920 22949 heartbeater.cc:503] Master 127.20.151.62:34821 was elected leader, sending a full tablet report...
I20240501 08:03:15.696072 23023 heartbeater.cc:503] Master 127.20.151.62:34821 was elected leader, sending a full tablet report...
I20240501 08:03:15.724488 21084 test_util.cc:268] Using random seed: -348838739
I20240501 08:03:15.746138 22747 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57710:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:03:15.748167 22747 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:03:15.792538 22914 tablet_service.cc:1467] Processing CreateTablet for tablet aefa79f68d754a96b24ca2e3dbe1c4fe (DEFAULT_TABLE table=test-workload [id=eb3ac07f22de43999369a9bcde18fe5e]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 08:03:15.792832 22913 tablet_service.cc:1467] Processing CreateTablet for tablet 32f307a2d3e740fb9b5254334ff05bb8 (DEFAULT_TABLE table=test-workload [id=eb3ac07f22de43999369a9bcde18fe5e]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 08:03:15.794126 22914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefa79f68d754a96b24ca2e3dbe1c4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:15.794178 22988 tablet_service.cc:1467] Processing CreateTablet for tablet aefa79f68d754a96b24ca2e3dbe1c4fe (DEFAULT_TABLE table=test-workload [id=eb3ac07f22de43999369a9bcde18fe5e]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 08:03:15.794641 22987 tablet_service.cc:1467] Processing CreateTablet for tablet 32f307a2d3e740fb9b5254334ff05bb8 (DEFAULT_TABLE table=test-workload [id=eb3ac07f22de43999369a9bcde18fe5e]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 08:03:15.795595 22988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefa79f68d754a96b24ca2e3dbe1c4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:15.796231 22987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f307a2d3e740fb9b5254334ff05bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:15.799944 22839 tablet_service.cc:1467] Processing CreateTablet for tablet aefa79f68d754a96b24ca2e3dbe1c4fe (DEFAULT_TABLE table=test-workload [id=eb3ac07f22de43999369a9bcde18fe5e]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 08:03:15.800575 22838 tablet_service.cc:1467] Processing CreateTablet for tablet 32f307a2d3e740fb9b5254334ff05bb8 (DEFAULT_TABLE table=test-workload [id=eb3ac07f22de43999369a9bcde18fe5e]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 08:03:15.801390 22839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefa79f68d754a96b24ca2e3dbe1c4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:15.802289 22913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f307a2d3e740fb9b5254334ff05bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:15.810050 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f307a2d3e740fb9b5254334ff05bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:15.827160 23043 tablet_bootstrap.cc:492] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c: Bootstrap starting.
I20240501 08:03:15.827502 23044 tablet_bootstrap.cc:492] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b: Bootstrap starting.
I20240501 08:03:15.834101 23043 tablet_bootstrap.cc:654] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:15.835005 23044 tablet_bootstrap.cc:654] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:15.836261 23045 tablet_bootstrap.cc:492] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e: Bootstrap starting.
I20240501 08:03:15.839797 23043 tablet_bootstrap.cc:492] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c: No bootstrap required, opened a new log
I20240501 08:03:15.840332 23043 ts_tablet_manager.cc:1367] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20240501 08:03:15.841805 23044 tablet_bootstrap.cc:492] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b: No bootstrap required, opened a new log
I20240501 08:03:15.842190 23044 ts_tablet_manager.cc:1367] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20240501 08:03:15.842307 23045 tablet_bootstrap.cc:654] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:15.843011 23043 raft_consensus.cc:348] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } }
I20240501 08:03:15.843808 23043 raft_consensus.cc:374] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:15.844127 23043 raft_consensus.cc:729] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d995ca92c4d247118c74bde8dfda226c, State: Initialized, Role: FOLLOWER
I20240501 08:03:15.844148 23044 raft_consensus.cc:348] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } }
I20240501 08:03:15.844962 23044 raft_consensus.cc:374] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:15.845412 23044 raft_consensus.cc:729] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb6f91c8b73428babc8d9ef41f4a42b, State: Initialized, Role: FOLLOWER
I20240501 08:03:15.844998 23043 consensus_queue.cc:260] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } }
I20240501 08:03:15.846258 23044 consensus_queue.cc:260] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } }
I20240501 08:03:15.847678 23045 tablet_bootstrap.cc:492] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e: No bootstrap required, opened a new log
I20240501 08:03:15.848002 23043 ts_tablet_manager.cc:1398] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240501 08:03:15.848306 23045 ts_tablet_manager.cc:1367] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240501 08:03:15.848937 23043 tablet_bootstrap.cc:492] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c: Bootstrap starting.
I20240501 08:03:15.854876 23043 tablet_bootstrap.cc:654] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:15.854277 23045 raft_consensus.cc:348] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } } peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } }
I20240501 08:03:15.855413 23045 raft_consensus.cc:374] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:15.855885 23045 raft_consensus.cc:729] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4877889d29a3461d9c9b25081fa3568e, State: Initialized, Role: FOLLOWER
I20240501 08:03:15.857024 23045 consensus_queue.cc:260] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } } peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } }
I20240501 08:03:15.864413 23045 ts_tablet_manager.cc:1398] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e: Time spent starting tablet: real 0.016s user 0.010s sys 0.000s
I20240501 08:03:15.864912 23044 ts_tablet_manager.cc:1398] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b: Time spent starting tablet: real 0.022s user 0.008s sys 0.013s
I20240501 08:03:15.866056 23044 tablet_bootstrap.cc:492] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b: Bootstrap starting.
I20240501 08:03:15.870918 23045 tablet_bootstrap.cc:492] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e: Bootstrap starting.
I20240501 08:03:15.871914 23044 tablet_bootstrap.cc:654] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:15.883292 23044 tablet_bootstrap.cc:492] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b: No bootstrap required, opened a new log
I20240501 08:03:15.883725 23044 ts_tablet_manager.cc:1367] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.002s
I20240501 08:03:15.883852 23043 tablet_bootstrap.cc:492] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c: No bootstrap required, opened a new log
I20240501 08:03:15.884359 23043 ts_tablet_manager.cc:1367] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c: Time spent bootstrapping tablet: real 0.036s user 0.023s sys 0.010s
I20240501 08:03:15.885634 23045 tablet_bootstrap.cc:654] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:15.885854 23044 raft_consensus.cc:348] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } } peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } }
I20240501 08:03:15.886454 23044 raft_consensus.cc:374] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:15.886675 23044 raft_consensus.cc:729] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb6f91c8b73428babc8d9ef41f4a42b, State: Initialized, Role: FOLLOWER
I20240501 08:03:15.886996 23043 raft_consensus.cc:348] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } } peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } }
I20240501 08:03:15.887290 23044 consensus_queue.cc:260] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } } peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } }
I20240501 08:03:15.887768 23043 raft_consensus.cc:374] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:15.888116 23043 raft_consensus.cc:729] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d995ca92c4d247118c74bde8dfda226c, State: Initialized, Role: FOLLOWER
I20240501 08:03:15.889719 23044 ts_tablet_manager.cc:1398] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240501 08:03:15.889024 23043 consensus_queue.cc:260] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } } peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } }
I20240501 08:03:15.891742 23043 ts_tablet_manager.cc:1398] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 08:03:15.894435 23045 tablet_bootstrap.cc:492] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e: No bootstrap required, opened a new log
I20240501 08:03:15.894840 23045 ts_tablet_manager.cc:1367] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.005s
I20240501 08:03:15.896878 23045 raft_consensus.cc:348] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } }
I20240501 08:03:15.897593 23045 raft_consensus.cc:374] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:15.897926 23045 raft_consensus.cc:729] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4877889d29a3461d9c9b25081fa3568e, State: Initialized, Role: FOLLOWER
I20240501 08:03:15.898538 23045 consensus_queue.cc:260] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } }
I20240501 08:03:15.900372 23045 ts_tablet_manager.cc:1398] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:03:15.921372 23049 raft_consensus.cc:482] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:15.921860 23049 raft_consensus.cc:504] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } }
I20240501 08:03:15.923738 23049 leader_election.cc:288] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4877889d29a3461d9c9b25081fa3568e (127.20.151.2:33869), 1cb6f91c8b73428babc8d9ef41f4a42b (127.20.151.1:36009)
I20240501 08:03:15.933620 22924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefa79f68d754a96b24ca2e3dbe1c4fe" candidate_uuid: "d995ca92c4d247118c74bde8dfda226c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4877889d29a3461d9c9b25081fa3568e" is_pre_election: true
I20240501 08:03:15.933768 22849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefa79f68d754a96b24ca2e3dbe1c4fe" candidate_uuid: "d995ca92c4d247118c74bde8dfda226c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" is_pre_election: true
I20240501 08:03:15.934672 22924 raft_consensus.cc:2454] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d995ca92c4d247118c74bde8dfda226c in term 0.
I20240501 08:03:15.934754 22849 raft_consensus.cc:2454] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d995ca92c4d247118c74bde8dfda226c in term 0.
I20240501 08:03:15.935763 22963 leader_election.cc:302] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4877889d29a3461d9c9b25081fa3568e, d995ca92c4d247118c74bde8dfda226c; no voters:
I20240501 08:03:15.936407 23049 raft_consensus.cc:2781] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:03:15.936758 23049 raft_consensus.cc:482] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:03:15.937002 23049 raft_consensus.cc:3037] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:15.941546 23049 raft_consensus.cc:504] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } }
I20240501 08:03:15.943042 23049 leader_election.cc:288] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [CANDIDATE]: Term 1 election: Requested vote from peers 4877889d29a3461d9c9b25081fa3568e (127.20.151.2:33869), 1cb6f91c8b73428babc8d9ef41f4a42b (127.20.151.1:36009)
I20240501 08:03:15.943848 22924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefa79f68d754a96b24ca2e3dbe1c4fe" candidate_uuid: "d995ca92c4d247118c74bde8dfda226c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4877889d29a3461d9c9b25081fa3568e"
I20240501 08:03:15.944073 22849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefa79f68d754a96b24ca2e3dbe1c4fe" candidate_uuid: "d995ca92c4d247118c74bde8dfda226c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b"
I20240501 08:03:15.944464 22924 raft_consensus.cc:3037] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:15.944583 22849 raft_consensus.cc:3037] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:15.951211 22924 raft_consensus.cc:2454] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d995ca92c4d247118c74bde8dfda226c in term 1.
I20240501 08:03:15.951215 22849 raft_consensus.cc:2454] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d995ca92c4d247118c74bde8dfda226c in term 1.
I20240501 08:03:15.952354 22963 leader_election.cc:302] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4877889d29a3461d9c9b25081fa3568e, d995ca92c4d247118c74bde8dfda226c; no voters:
I20240501 08:03:15.952965 23049 raft_consensus.cc:2781] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:15.953948 23049 raft_consensus.cc:686] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 1 LEADER]: Becoming Leader. State: Replica: d995ca92c4d247118c74bde8dfda226c, State: Running, Role: LEADER
I20240501 08:03:15.954655 23049 consensus_queue.cc:237] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } }
I20240501 08:03:15.962389 22746 catalog_manager.cc:5484] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c reported cstate change: term changed from 0 to 1, leader changed from <none> to d995ca92c4d247118c74bde8dfda226c (127.20.151.3). New cstate: current_term: 1 leader_uuid: "d995ca92c4d247118c74bde8dfda226c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:16.067270 23050 raft_consensus.cc:482] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:03:16.067703 23050 raft_consensus.cc:504] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } } peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } }
I20240501 08:03:16.069482 23050 leader_election.cc:288] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4877889d29a3461d9c9b25081fa3568e (127.20.151.2:33869), d995ca92c4d247118c74bde8dfda226c (127.20.151.3:46073)
I20240501 08:03:16.080061 22924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f307a2d3e740fb9b5254334ff05bb8" candidate_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4877889d29a3461d9c9b25081fa3568e" is_pre_election: true
I20240501 08:03:16.080150 22998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f307a2d3e740fb9b5254334ff05bb8" candidate_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d995ca92c4d247118c74bde8dfda226c" is_pre_election: true
I20240501 08:03:16.080883 22924 raft_consensus.cc:2454] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb6f91c8b73428babc8d9ef41f4a42b in term 0.
I20240501 08:03:16.080891 22998 raft_consensus.cc:2454] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb6f91c8b73428babc8d9ef41f4a42b in term 0.
I20240501 08:03:16.082146 22814 leader_election.cc:302] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb6f91c8b73428babc8d9ef41f4a42b, 4877889d29a3461d9c9b25081fa3568e; no voters:
I20240501 08:03:16.082808 23050 raft_consensus.cc:2781] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:03:16.083056 23050 raft_consensus.cc:482] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:03:16.083253 23050 raft_consensus.cc:3037] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:16.087355 23050 raft_consensus.cc:504] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } } peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } }
I20240501 08:03:16.088693 23050 leader_election.cc:288] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [CANDIDATE]: Term 1 election: Requested vote from peers 4877889d29a3461d9c9b25081fa3568e (127.20.151.2:33869), d995ca92c4d247118c74bde8dfda226c (127.20.151.3:46073)
I20240501 08:03:16.089376 22924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f307a2d3e740fb9b5254334ff05bb8" candidate_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4877889d29a3461d9c9b25081fa3568e"
I20240501 08:03:16.089618 22998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f307a2d3e740fb9b5254334ff05bb8" candidate_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d995ca92c4d247118c74bde8dfda226c"
I20240501 08:03:16.089848 22924 raft_consensus.cc:3037] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:16.090099 22998 raft_consensus.cc:3037] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:16.094305 22924 raft_consensus.cc:2454] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb6f91c8b73428babc8d9ef41f4a42b in term 1.
I20240501 08:03:16.094985 22998 raft_consensus.cc:2454] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb6f91c8b73428babc8d9ef41f4a42b in term 1.
I20240501 08:03:16.095181 22814 leader_election.cc:302] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb6f91c8b73428babc8d9ef41f4a42b, 4877889d29a3461d9c9b25081fa3568e; no voters:
I20240501 08:03:16.096000 23050 raft_consensus.cc:2781] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:16.096920 23050 raft_consensus.cc:686] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 1 LEADER]: Becoming Leader. State: Replica: 1cb6f91c8b73428babc8d9ef41f4a42b, State: Running, Role: LEADER
I20240501 08:03:16.097498 23050 consensus_queue.cc:237] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } } peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } }
I20240501 08:03:16.103619 22746 catalog_manager.cc:5484] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb6f91c8b73428babc8d9ef41f4a42b (127.20.151.1). New cstate: current_term: 1 leader_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:16.341902 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:16.343709 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:16.344012 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:16.410136 23049 consensus_queue.cc:1035] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:16.421825 23049 consensus_queue.cc:1035] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:16.538247 23050 consensus_queue.cc:1035] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:16.547788 23061 consensus_queue.cc:1035] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:03:18.345023 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:18.347633 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:18.348085 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:20.349130 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:20.350983 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:20.351253 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:22.351995 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:22.353679 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:22.353938 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:24.354972 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:24.356923 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:24.357211 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:26.357980 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:26.359650 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:26.359906 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:27.228591 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:27.236670 23081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:27.237768 23082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:27.238613 23084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:27.240288 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:27.241184 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:27.241400 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:27.241581 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550607241567 us; error 0 us; skew 500 ppm
I20240501 08:03:27.242245 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:27.250480 21084 webserver.cc:460] Webserver started at http://127.20.151.4:41387/ using document root <none> and password file <none>
I20240501 08:03:27.250986 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:27.251194 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:27.251506 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:27.252885 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-3-root/instance:
uuid: "11cd6e51689f4fb8820756f2389f0a4d"
format_stamp: "Formatted at 2024-05-01 08:03:27 on dist-test-slave-pwkb"
I20240501 08:03:27.257946 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20240501 08:03:27.261257 23089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:27.262104 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:03:27.262341 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-3-root
uuid: "11cd6e51689f4fb8820756f2389f0a4d"
format_stamp: "Formatted at 2024-05-01 08:03:27 on dist-test-slave-pwkb"
I20240501 08:03:27.262575 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:27.289528 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:27.290803 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:27.292240 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:03:27.294654 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:03:27.294837 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:27.295128 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:03:27.295292 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:27.334620 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.4:33933
I20240501 08:03:27.334796 23151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.4:33933 every 8 connection(s)
I20240501 08:03:27.349750 23152 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34821
I20240501 08:03:27.350121 23152 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:27.350850 23152 heartbeater.cc:511] Master 127.20.151.62:34821 requested a full tablet report, sending...
I20240501 08:03:27.352612 22747 ts_manager.cc:194] Registered new tserver with Master: 11cd6e51689f4fb8820756f2389f0a4d (127.20.151.4:33933)
I20240501 08:03:27.353994 22747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41264
I20240501 08:03:27.399994 22998 consensus_queue.cc:237] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: true } }
I20240501 08:03:27.407727 22849 raft_consensus.cc:1261] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [term 1 FOLLOWER]: Refusing update from remote peer d995ca92c4d247118c74bde8dfda226c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:03:27.408766 22924 raft_consensus.cc:1261] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 1 FOLLOWER]: Refusing update from remote peer d995ca92c4d247118c74bde8dfda226c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:03:27.409591 23159 consensus_queue.cc:1035] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:03:27.410651 23078 consensus_queue.cc:1035] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:03:27.419209 23078 raft_consensus.cc:2932] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 11cd6e51689f4fb8820756f2389f0a4d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: true } } }
W20240501 08:03:27.420934 22963 consensus_peers.cc:485] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c -> Peer 11cd6e51689f4fb8820756f2389f0a4d (127.20.151.4:33933): Couldn't send request to peer 11cd6e51689f4fb8820756f2389f0a4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aefa79f68d754a96b24ca2e3dbe1c4fe. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:03:27.421765 22849 raft_consensus.cc:2932] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 11cd6e51689f4fb8820756f2389f0a4d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: true } } }
I20240501 08:03:27.422752 22924 raft_consensus.cc:2932] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 11cd6e51689f4fb8820756f2389f0a4d (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: true } } }
I20240501 08:03:27.432302 22747 catalog_manager.cc:5484] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c reported cstate change: config changed from index -1 to 2, NON_VOTER 11cd6e51689f4fb8820756f2389f0a4d (127.20.151.4) added. New cstate: current_term: 1 leader_uuid: "d995ca92c4d247118c74bde8dfda226c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:03:27.850327 23169 ts_tablet_manager.cc:902] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d: Initiating tablet copy from peer d995ca92c4d247118c74bde8dfda226c (127.20.151.3:46073)
I20240501 08:03:27.854240 23169 tablet_copy_client.cc:308] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.3:46073
I20240501 08:03:27.863520 23008 tablet_copy_service.cc:140] P d995ca92c4d247118c74bde8dfda226c: Received BeginTabletCopySession request for tablet aefa79f68d754a96b24ca2e3dbe1c4fe from peer 11cd6e51689f4fb8820756f2389f0a4d ({username='slave'} at 127.0.0.1:46518)
I20240501 08:03:27.864126 23008 tablet_copy_service.cc:161] P d995ca92c4d247118c74bde8dfda226c: Beginning new tablet copy session on tablet aefa79f68d754a96b24ca2e3dbe1c4fe from peer 11cd6e51689f4fb8820756f2389f0a4d at {username='slave'} at 127.0.0.1:46518: session id = 11cd6e51689f4fb8820756f2389f0a4d-aefa79f68d754a96b24ca2e3dbe1c4fe
I20240501 08:03:27.871322 23008 tablet_copy_source_session.cc:205] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:03:27.873854 23169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefa79f68d754a96b24ca2e3dbe1c4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:27.885699 23169 tablet_copy_client.cc:783] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d: tablet copy: Starting download of 0 data blocks...
I20240501 08:03:27.886210 23169 tablet_copy_client.cc:647] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d: tablet copy: Starting download of 1 WAL segments...
I20240501 08:03:27.889158 23169 tablet_copy_client.cc:523] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:03:27.895143 23169 tablet_bootstrap.cc:492] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d: Bootstrap starting.
I20240501 08:03:27.908716 23169 tablet_bootstrap.cc:492] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:03:27.909353 23169 tablet_bootstrap.cc:492] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d: Bootstrap complete.
I20240501 08:03:27.909999 23169 ts_tablet_manager.cc:1367] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.005s
I20240501 08:03:27.912065 23169 raft_consensus.cc:348] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: true } }
I20240501 08:03:27.912760 23169 raft_consensus.cc:729] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 11cd6e51689f4fb8820756f2389f0a4d, State: Initialized, Role: LEARNER
I20240501 08:03:27.913331 23169 consensus_queue.cc:260] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: true } }
I20240501 08:03:27.915405 23152 heartbeater.cc:503] Master 127.20.151.62:34821 was elected leader, sending a full tablet report...
I20240501 08:03:27.916664 23169 ts_tablet_manager.cc:1398] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 08:03:27.918268 23008 tablet_copy_service.cc:342] P d995ca92c4d247118c74bde8dfda226c: Request end of tablet copy session 11cd6e51689f4fb8820756f2389f0a4d-aefa79f68d754a96b24ca2e3dbe1c4fe received from {username='slave'} at 127.0.0.1:46518
I20240501 08:03:27.918661 23008 tablet_copy_service.cc:433] P d995ca92c4d247118c74bde8dfda226c: ending tablet copy session 11cd6e51689f4fb8820756f2389f0a4d-aefa79f68d754a96b24ca2e3dbe1c4fe on tablet aefa79f68d754a96b24ca2e3dbe1c4fe with peer 11cd6e51689f4fb8820756f2389f0a4d
I20240501 08:03:28.354130 23127 raft_consensus.cc:1203] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:03:28.360582 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:28.362851 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:28.363291 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:28.875120 23177 raft_consensus.cc:1050] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c: attempting to promote NON_VOTER 11cd6e51689f4fb8820756f2389f0a4d to VOTER
I20240501 08:03:28.877090 23177 consensus_queue.cc:237] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false } }
I20240501 08:03:28.883102 23127 raft_consensus.cc:1261] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d [term 1 LEARNER]: Refusing update from remote peer d995ca92c4d247118c74bde8dfda226c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:03:28.883417 22849 raft_consensus.cc:1261] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [term 1 FOLLOWER]: Refusing update from remote peer d995ca92c4d247118c74bde8dfda226c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:03:28.884907 23179 consensus_queue.cc:1035] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:03:28.885778 23177 consensus_queue.cc:1035] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:03:28.893698 22924 raft_consensus.cc:1261] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 1 FOLLOWER]: Refusing update from remote peer d995ca92c4d247118c74bde8dfda226c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:03:28.895206 23179 consensus_queue.cc:1035] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:03:28.898506 23181 raft_consensus.cc:2932] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 11cd6e51689f4fb8820756f2389f0a4d (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false } } }
I20240501 08:03:28.900255 23127 raft_consensus.cc:2932] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 11cd6e51689f4fb8820756f2389f0a4d (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false } } }
I20240501 08:03:28.904173 22924 raft_consensus.cc:2932] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 11cd6e51689f4fb8820756f2389f0a4d (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false } } }
I20240501 08:03:28.909216 22849 raft_consensus.cc:2932] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 11cd6e51689f4fb8820756f2389f0a4d (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false } } }
I20240501 08:03:28.912314 22745 catalog_manager.cc:5484] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c reported cstate change: config changed from index 2 to 3, 11cd6e51689f4fb8820756f2389f0a4d (127.20.151.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d995ca92c4d247118c74bde8dfda226c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cb6f91c8b73428babc8d9ef41f4a42b" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36009 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:03:28.941454 22998 consensus_queue.cc:237] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false } }
I20240501 08:03:28.945917 23127 raft_consensus.cc:1261] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d [term 1 FOLLOWER]: Refusing update from remote peer d995ca92c4d247118c74bde8dfda226c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:03:28.946416 22924 raft_consensus.cc:1261] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 1 FOLLOWER]: Refusing update from remote peer d995ca92c4d247118c74bde8dfda226c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:03:28.947002 23177 consensus_queue.cc:1035] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:03:28.947875 23179 consensus_queue.cc:1035] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:03:28.953940 23181 raft_consensus.cc:2932] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1cb6f91c8b73428babc8d9ef41f4a42b (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false } } }
I20240501 08:03:28.955623 22924 raft_consensus.cc:2932] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1cb6f91c8b73428babc8d9ef41f4a42b (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false } } }
I20240501 08:03:28.955477 23127 raft_consensus.cc:2932] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1cb6f91c8b73428babc8d9ef41f4a42b (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false } } }
I20240501 08:03:28.961817 22731 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aefa79f68d754a96b24ca2e3dbe1c4fe with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:03:28.964169 22744 catalog_manager.cc:5484] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c reported cstate change: config changed from index 3 to 4, VOTER 1cb6f91c8b73428babc8d9ef41f4a42b (127.20.151.1) evicted. New cstate: current_term: 1 leader_uuid: "d995ca92c4d247118c74bde8dfda226c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4877889d29a3461d9c9b25081fa3568e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d995ca92c4d247118c74bde8dfda226c" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11cd6e51689f4fb8820756f2389f0a4d" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:03:28.973415 22839 tablet_service.cc:1514] Processing DeleteTablet for tablet aefa79f68d754a96b24ca2e3dbe1c4fe with delete_type TABLET_DATA_TOMBSTONED (TS 1cb6f91c8b73428babc8d9ef41f4a42b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46686
I20240501 08:03:28.975567 23191 tablet_replica.cc:330] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b: stopping tablet replica
I20240501 08:03:28.976315 23191 raft_consensus.cc:2229] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:28.976843 23191 raft_consensus.cc:2258] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:28.979225 23191 ts_tablet_manager.cc:1875] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:03:28.988653 23191 ts_tablet_manager.cc:1888] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:03:28.988950 23191 log.cc:1196] T aefa79f68d754a96b24ca2e3dbe1c4fe P 1cb6f91c8b73428babc8d9ef41f4a42b: Deleting WAL directory at /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550573434241-21084-0/minicluster-data/ts-0-root/wals/aefa79f68d754a96b24ca2e3dbe1c4fe
I20240501 08:03:28.990144 22731 catalog_manager.cc:4830] TS 1cb6f91c8b73428babc8d9ef41f4a42b (127.20.151.1:36009): tablet aefa79f68d754a96b24ca2e3dbe1c4fe (table test-workload [id=eb3ac07f22de43999369a9bcde18fe5e]) successfully deleted
I20240501 08:03:30.364182 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:30.366159 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:30.366437 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:32.367303 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:32.369094 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:32.369350 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:34.370352 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:34.372347 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:34.372658 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:36.373620 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:36.375440 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:36.375730 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:38.376829 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:38.379895 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:38.380371 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:40.381314 22800 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:03:40.383203 22800 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:03:40.383473 22800 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:41.050016 21084 tablet_server.cc:178] TabletServer@127.20.151.1:0 shutting down...
I20240501 08:03:41.075218 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:03:41.075865 21084 tablet_replica.cc:330] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b: stopping tablet replica
I20240501 08:03:41.076386 21084 raft_consensus.cc:2229] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:41.077243 21084 raft_consensus.cc:2258] T 32f307a2d3e740fb9b5254334ff05bb8 P 1cb6f91c8b73428babc8d9ef41f4a42b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:41.098618 21084 tablet_server.cc:194] TabletServer@127.20.151.1:0 shutdown complete.
I20240501 08:03:41.110142 21084 tablet_server.cc:178] TabletServer@127.20.151.2:0 shutting down...
I20240501 08:03:41.132438 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:03:41.133280 21084 tablet_replica.cc:330] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e: stopping tablet replica
I20240501 08:03:41.134004 21084 raft_consensus.cc:2229] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:41.134876 21084 raft_consensus.cc:2258] T aefa79f68d754a96b24ca2e3dbe1c4fe P 4877889d29a3461d9c9b25081fa3568e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:41.136785 21084 tablet_replica.cc:330] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e: stopping tablet replica
I20240501 08:03:41.137209 21084 raft_consensus.cc:2229] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:41.137581 21084 raft_consensus.cc:2258] T 32f307a2d3e740fb9b5254334ff05bb8 P 4877889d29a3461d9c9b25081fa3568e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:41.157641 21084 tablet_server.cc:194] TabletServer@127.20.151.2:0 shutdown complete.
I20240501 08:03:41.167701 21084 tablet_server.cc:178] TabletServer@127.20.151.3:0 shutting down...
I20240501 08:03:41.188701 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:03:41.189348 21084 tablet_replica.cc:330] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c: stopping tablet replica
I20240501 08:03:41.189945 21084 raft_consensus.cc:2229] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:41.190369 21084 raft_consensus.cc:2258] T 32f307a2d3e740fb9b5254334ff05bb8 P d995ca92c4d247118c74bde8dfda226c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:41.192198 21084 tablet_replica.cc:330] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c: stopping tablet replica
I20240501 08:03:41.192628 21084 raft_consensus.cc:2229] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:41.193456 21084 raft_consensus.cc:2258] T aefa79f68d754a96b24ca2e3dbe1c4fe P d995ca92c4d247118c74bde8dfda226c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:41.215009 21084 tablet_server.cc:194] TabletServer@127.20.151.3:0 shutdown complete.
I20240501 08:03:41.225311 21084 tablet_server.cc:178] TabletServer@127.20.151.4:0 shutting down...
I20240501 08:03:41.249459 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:03:41.250154 21084 tablet_replica.cc:330] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d: stopping tablet replica
I20240501 08:03:41.250705 21084 raft_consensus.cc:2229] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:03:41.251323 21084 raft_consensus.cc:2258] T aefa79f68d754a96b24ca2e3dbe1c4fe P 11cd6e51689f4fb8820756f2389f0a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:41.269634 21084 tablet_server.cc:194] TabletServer@127.20.151.4:0 shutdown complete.
I20240501 08:03:41.278879 21084 master.cc:537] Master@127.20.151.62:34821 shutting down...
I20240501 08:03:41.295598 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:41.296157 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:41.296461 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P e91c8826265b48fa8e9c49fe52eca850: stopping tablet replica
I20240501 08:03:41.313777 21084 master.cc:552] Master@127.20.151.62:34821 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (27145 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240501 08:03:41.348762 21084 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.151.62:41537
I20240501 08:03:41.349836 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:41.355952 23196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:41.356902 23197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:41.357805 23199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:41.358325 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:41.359182 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:41.359388 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:41.359534 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550621359516 us; error 0 us; skew 500 ppm
I20240501 08:03:41.360013 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:41.362553 21084 webserver.cc:460] Webserver started at http://127.20.151.62:42275/ using document root <none> and password file <none>
I20240501 08:03:41.363226 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:41.363420 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:41.363649 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:41.364781 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714550573434241-21084-0/minicluster-data/master-0-root/instance:
uuid: "19854f966e144e5bb3dabad07f191093"
format_stamp: "Formatted at 2024-05-01 08:03:41 on dist-test-slave-pwkb"
I20240501 08:03:41.369045 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240501 08:03:41.372159 23204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:41.372884 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:03:41.373126 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714550573434241-21084-0/minicluster-data/master-0-root
uuid: "19854f966e144e5bb3dabad07f191093"
format_stamp: "Formatted at 2024-05-01 08:03:41 on dist-test-slave-pwkb"
I20240501 08:03:41.373375 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714550573434241-21084-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714550573434241-21084-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714550573434241-21084-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:41.392058 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:41.393297 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:41.430447 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:41537
I20240501 08:03:41.430523 23255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:41537 every 8 connection(s)
I20240501 08:03:41.434304 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:41.444106 23256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093: Bootstrap starting.
I20240501 08:03:41.448055 23256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:41.452152 23256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093: No bootstrap required, opened a new log
I20240501 08:03:41.454087 23256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19854f966e144e5bb3dabad07f191093" member_type: VOTER }
I20240501 08:03:41.454524 23256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:41.454694 23256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19854f966e144e5bb3dabad07f191093, State: Initialized, Role: FOLLOWER
I20240501 08:03:41.455204 23256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19854f966e144e5bb3dabad07f191093" member_type: VOTER }
I20240501 08:03:41.455654 23256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:03:41.455838 23256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:03:41.456051 23256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:41.459981 23256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19854f966e144e5bb3dabad07f191093" member_type: VOTER }
I20240501 08:03:41.460415 23256 leader_election.cc:302] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19854f966e144e5bb3dabad07f191093; no voters:
I20240501 08:03:41.461434 23256 leader_election.cc:288] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:03:41.461838 23259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:41.463066 23259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [term 1 LEADER]: Becoming Leader. State: Replica: 19854f966e144e5bb3dabad07f191093, State: Running, Role: LEADER
I20240501 08:03:41.463725 23259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19854f966e144e5bb3dabad07f191093" member_type: VOTER }
I20240501 08:03:41.464227 23256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:03:41.466943 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19854f966e144e5bb3dabad07f191093. Latest consensus state: current_term: 1 leader_uuid: "19854f966e144e5bb3dabad07f191093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19854f966e144e5bb3dabad07f191093" member_type: VOTER } }
I20240501 08:03:41.466886 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19854f966e144e5bb3dabad07f191093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19854f966e144e5bb3dabad07f191093" member_type: VOTER } }
I20240501 08:03:41.467665 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [sys.catalog]: This master's current role is: LEADER
I20240501 08:03:41.467763 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [sys.catalog]: This master's current role is: LEADER
I20240501 08:03:41.470539 23265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:03:41.476588 23265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:03:41.479534 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:03:41.484884 23265 catalog_manager.cc:1347] Generated new cluster ID: 49fd521ffc074557b1ed3729e8bb6a8d
I20240501 08:03:41.485175 23265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:03:41.504016 23265 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:03:41.505208 23265 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:03:41.522238 23265 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093: Generated new TSK 0
I20240501 08:03:41.522816 23265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:03:41.544939 21084 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.000180876s
I20240501 08:03:53.638198 21084 master.cc:537] Master@127.20.151.62:41537 shutting down...
I20240501 08:03:53.660017 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:03:53.660866 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:03:53.661216 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P 19854f966e144e5bb3dabad07f191093: stopping tablet replica
I20240501 08:03:53.680357 21084 master.cc:552] Master@127.20.151.62:41537 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12362 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240501 08:03:53.711212 21084 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.151.62:41323
I20240501 08:03:53.712318 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:53.718070 23279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:53.719122 23280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:53.719441 23282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:53.720292 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:53.720984 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:53.721172 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:53.721308 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550633721290 us; error 0 us; skew 500 ppm
I20240501 08:03:53.721812 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:53.729660 21084 webserver.cc:460] Webserver started at http://127.20.151.62:45885/ using document root <none> and password file <none>
I20240501 08:03:53.730226 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:53.730399 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:53.730686 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:53.731884 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/master-0-root/instance:
uuid: "1f06dcfd38f14d93a466c9736f4d3974"
format_stamp: "Formatted at 2024-05-01 08:03:53 on dist-test-slave-pwkb"
I20240501 08:03:53.736580 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240501 08:03:53.739943 23287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:53.740607 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 08:03:53.740862 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/master-0-root
uuid: "1f06dcfd38f14d93a466c9736f4d3974"
format_stamp: "Formatted at 2024-05-01 08:03:53 on dist-test-slave-pwkb"
I20240501 08:03:53.741119 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:53.784206 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:53.785470 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:53.828332 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:41323
I20240501 08:03:53.828418 23339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:41323 every 8 connection(s)
I20240501 08:03:53.832795 23340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:03:53.844213 23340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974: Bootstrap starting.
I20240501 08:03:53.848635 23340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974: Neither blocks nor log segments found. Creating new log.
I20240501 08:03:53.852804 23340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974: No bootstrap required, opened a new log
I20240501 08:03:53.854866 23340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f06dcfd38f14d93a466c9736f4d3974" member_type: VOTER }
I20240501 08:03:53.855404 23340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:03:53.855589 23340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f06dcfd38f14d93a466c9736f4d3974, State: Initialized, Role: FOLLOWER
I20240501 08:03:53.856074 23340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f06dcfd38f14d93a466c9736f4d3974" member_type: VOTER }
I20240501 08:03:53.856462 23340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:03:53.856637 23340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:03:53.856848 23340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:03:53.860872 23340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f06dcfd38f14d93a466c9736f4d3974" member_type: VOTER }
I20240501 08:03:53.861302 23340 leader_election.cc:302] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f06dcfd38f14d93a466c9736f4d3974; no voters:
I20240501 08:03:53.862362 23340 leader_election.cc:288] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:03:53.862732 23343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:03:53.863915 23343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [term 1 LEADER]: Becoming Leader. State: Replica: 1f06dcfd38f14d93a466c9736f4d3974, State: Running, Role: LEADER
I20240501 08:03:53.864620 23343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f06dcfd38f14d93a466c9736f4d3974" member_type: VOTER }
I20240501 08:03:53.865269 23340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:03:53.867774 23344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f06dcfd38f14d93a466c9736f4d3974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f06dcfd38f14d93a466c9736f4d3974" member_type: VOTER } }
I20240501 08:03:53.867801 23345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f06dcfd38f14d93a466c9736f4d3974. Latest consensus state: current_term: 1 leader_uuid: "1f06dcfd38f14d93a466c9736f4d3974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f06dcfd38f14d93a466c9736f4d3974" member_type: VOTER } }
I20240501 08:03:53.868518 23344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [sys.catalog]: This master's current role is: LEADER
I20240501 08:03:53.869408 23345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [sys.catalog]: This master's current role is: LEADER
I20240501 08:03:53.874946 23350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:03:53.879988 23350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:03:53.881278 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:03:53.888007 23350 catalog_manager.cc:1347] Generated new cluster ID: f90a31e3765c4621a01b665f689164cd
I20240501 08:03:53.888224 23350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:03:53.922521 23350 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:03:53.923751 23350 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:03:53.940097 23350 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974: Generated new TSK 0
I20240501 08:03:53.940636 23350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:03:53.947418 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:53.954680 23361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:53.955637 23362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:53.957374 21084 server_base.cc:1034] running on GCE node
W20240501 08:03:53.957911 23364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:53.958671 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:53.958879 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:53.959038 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550633959020 us; error 0 us; skew 500 ppm
I20240501 08:03:53.959550 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:53.962000 21084 webserver.cc:460] Webserver started at http://127.20.151.1:38935/ using document root <none> and password file <none>
I20240501 08:03:53.962435 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:53.962620 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:53.962899 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:53.964151 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-0-root/instance:
uuid: "2fe7a38939034732b35d51639a2be848"
format_stamp: "Formatted at 2024-05-01 08:03:53 on dist-test-slave-pwkb"
I20240501 08:03:53.968828 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240501 08:03:53.972031 23369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:53.972755 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:03:53.973004 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-0-root
uuid: "2fe7a38939034732b35d51639a2be848"
format_stamp: "Formatted at 2024-05-01 08:03:53 on dist-test-slave-pwkb"
I20240501 08:03:53.973274 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:53.997723 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:53.998852 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:54.000270 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:03:54.002585 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:03:54.002815 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:54.003057 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:03:54.003225 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:54.042817 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:38583
I20240501 08:03:54.042881 23431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:38583 every 8 connection(s)
I20240501 08:03:54.049185 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:54.056425 23436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:54.057590 23437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:54.061393 23439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:54.062449 23432 heartbeater.cc:347] Connected to a master server at 127.20.151.62:41323
I20240501 08:03:54.062801 23432 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:54.063529 23432 heartbeater.cc:511] Master 127.20.151.62:41323 requested a full tablet report, sending...
I20240501 08:03:54.065634 21084 server_base.cc:1034] running on GCE node
I20240501 08:03:54.065754 23305 ts_manager.cc:194] Registered new tserver with Master: 2fe7a38939034732b35d51639a2be848 (127.20.151.1:38583)
I20240501 08:03:54.066553 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:54.066752 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:54.066918 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550634066902 us; error 0 us; skew 500 ppm
I20240501 08:03:54.067454 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:54.068002 23305 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55554
I20240501 08:03:54.070308 21084 webserver.cc:460] Webserver started at http://127.20.151.2:34115/ using document root <none> and password file <none>
I20240501 08:03:54.070762 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:54.070919 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:54.071158 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:54.072288 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-1-root/instance:
uuid: "6de9af021fef4de286ec19baf3da6518"
format_stamp: "Formatted at 2024-05-01 08:03:54 on dist-test-slave-pwkb"
I20240501 08:03:54.076551 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:03:54.079664 23444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:54.080444 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:03:54.080699 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-1-root
uuid: "6de9af021fef4de286ec19baf3da6518"
format_stamp: "Formatted at 2024-05-01 08:03:54 on dist-test-slave-pwkb"
I20240501 08:03:54.080937 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:54.090708 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:54.091781 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:54.093032 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:03:54.095182 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:03:54.095382 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:54.095594 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:03:54.095746 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:54.133033 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:42849
I20240501 08:03:54.133138 23506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:42849 every 8 connection(s)
I20240501 08:03:54.137593 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:03:54.144603 23510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:03:54.146224 23511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:54.148934 21084 server_base.cc:1034] running on GCE node
W20240501 08:03:54.149717 23513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:03:54.150214 23507 heartbeater.cc:347] Connected to a master server at 127.20.151.62:41323
I20240501 08:03:54.150629 23507 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:54.150714 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:03:54.150997 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:03:54.151196 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550634151178 us; error 0 us; skew 500 ppm
I20240501 08:03:54.151522 23507 heartbeater.cc:511] Master 127.20.151.62:41323 requested a full tablet report, sending...
I20240501 08:03:54.151985 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:03:54.153566 23305 ts_manager.cc:194] Registered new tserver with Master: 6de9af021fef4de286ec19baf3da6518 (127.20.151.2:42849)
I20240501 08:03:54.154942 23305 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55568
I20240501 08:03:54.155475 21084 webserver.cc:460] Webserver started at http://127.20.151.3:35531/ using document root <none> and password file <none>
I20240501 08:03:54.156006 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:03:54.156200 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:03:54.156458 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:03:54.157620 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-2-root/instance:
uuid: "9b94ea2e8b964d338abe30f74b4f443f"
format_stamp: "Formatted at 2024-05-01 08:03:54 on dist-test-slave-pwkb"
I20240501 08:03:54.161669 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 08:03:54.164739 23518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:54.165462 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240501 08:03:54.165786 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-2-root
uuid: "9b94ea2e8b964d338abe30f74b4f443f"
format_stamp: "Formatted at 2024-05-01 08:03:54 on dist-test-slave-pwkb"
I20240501 08:03:54.166028 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550573434241-21084-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:03:54.176699 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:03:54.177793 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:03:54.179127 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:03:54.181638 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:03:54.181854 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:54.182077 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:03:54.182212 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:03:54.219949 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:38167
I20240501 08:03:54.220091 23580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:38167 every 8 connection(s)
I20240501 08:03:54.232625 23581 heartbeater.cc:347] Connected to a master server at 127.20.151.62:41323
I20240501 08:03:54.232951 23581 heartbeater.cc:464] Registering TS with master...
I20240501 08:03:54.233577 23581 heartbeater.cc:511] Master 127.20.151.62:41323 requested a full tablet report, sending...
I20240501 08:03:54.235157 23305 ts_manager.cc:194] Registered new tserver with Master: 9b94ea2e8b964d338abe30f74b4f443f (127.20.151.3:38167)
I20240501 08:03:54.236268 21084 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013198706s
I20240501 08:03:54.236383 23305 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55570
I20240501 08:03:55.070703 23432 heartbeater.cc:503] Master 127.20.151.62:41323 was elected leader, sending a full tablet report...
I20240501 08:03:55.157171 23507 heartbeater.cc:503] Master 127.20.151.62:41323 was elected leader, sending a full tablet report...
I20240501 08:03:55.238683 23581 heartbeater.cc:503] Master 127.20.151.62:41323 was elected leader, sending a full tablet report...
I20240501 08:03:55.880862 23358 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:57.881840 23358 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:03:59.882807 23358 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:01.883695 23358 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:03.884997 23358 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:05.886135 23358 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:06.328703 21084 tablet_server.cc:178] TabletServer@127.20.151.1:0 shutting down...
I20240501 08:04:06.356429 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:06.372810 21084 tablet_server.cc:194] TabletServer@127.20.151.1:0 shutdown complete.
I20240501 08:04:06.382128 21084 tablet_server.cc:178] TabletServer@127.20.151.2:0 shutting down...
I20240501 08:04:06.405598 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:06.421134 21084 tablet_server.cc:194] TabletServer@127.20.151.2:0 shutdown complete.
I20240501 08:04:06.429229 21084 tablet_server.cc:178] TabletServer@127.20.151.3:0 shutting down...
I20240501 08:04:06.450134 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:06.465175 21084 tablet_server.cc:194] TabletServer@127.20.151.3:0 shutdown complete.
I20240501 08:04:06.473431 21084 master.cc:537] Master@127.20.151.62:41323 shutting down...
I20240501 08:04:06.491356 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:04:06.492046 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:06.492405 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1f06dcfd38f14d93a466c9736f4d3974: stopping tablet replica
I20240501 08:04:06.509871 21084 master.cc:552] Master@127.20.151.62:41323 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12826 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240501 08:04:06.537314 21084 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.151.62:43005
I20240501 08:04:06.538412 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:06.544348 23586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:06.545138 23587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:06.546131 23589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:06.547015 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:06.547694 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:06.547881 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:06.548020 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550646548001 us; error 0 us; skew 500 ppm
I20240501 08:04:06.548513 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:06.556277 21084 webserver.cc:460] Webserver started at http://127.20.151.62:45161/ using document root <none> and password file <none>
I20240501 08:04:06.556718 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:06.556874 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:06.557126 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:06.558319 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/master-0-root/instance:
uuid: "c71a4ef568084fd1a0cc013c4448536d"
format_stamp: "Formatted at 2024-05-01 08:04:06 on dist-test-slave-pwkb"
I20240501 08:04:06.562489 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240501 08:04:06.565455 23594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:06.566119 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:04:06.566349 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/master-0-root
uuid: "c71a4ef568084fd1a0cc013c4448536d"
format_stamp: "Formatted at 2024-05-01 08:04:06 on dist-test-slave-pwkb"
I20240501 08:04:06.566576 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:06.595791 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:06.596966 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:06.632395 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:43005
I20240501 08:04:06.632488 23645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:43005 every 8 connection(s)
I20240501 08:04:06.636266 23646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:06.646704 23646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d: Bootstrap starting.
I20240501 08:04:06.650748 23646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:06.654703 23646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d: No bootstrap required, opened a new log
I20240501 08:04:06.656563 23646 raft_consensus.cc:348] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71a4ef568084fd1a0cc013c4448536d" member_type: VOTER }
I20240501 08:04:06.657020 23646 raft_consensus.cc:374] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:06.657197 23646 raft_consensus.cc:729] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71a4ef568084fd1a0cc013c4448536d, State: Initialized, Role: FOLLOWER
I20240501 08:04:06.657757 23646 consensus_queue.cc:260] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71a4ef568084fd1a0cc013c4448536d" member_type: VOTER }
I20240501 08:04:06.658174 23646 raft_consensus.cc:388] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:04:06.658355 23646 raft_consensus.cc:482] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:04:06.658578 23646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:06.662382 23646 raft_consensus.cc:504] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71a4ef568084fd1a0cc013c4448536d" member_type: VOTER }
I20240501 08:04:06.662788 23646 leader_election.cc:302] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c71a4ef568084fd1a0cc013c4448536d; no voters:
I20240501 08:04:06.663760 23646 leader_election.cc:288] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:04:06.664098 23649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:04:06.665294 23649 raft_consensus.cc:686] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [term 1 LEADER]: Becoming Leader. State: Replica: c71a4ef568084fd1a0cc013c4448536d, State: Running, Role: LEADER
I20240501 08:04:06.665876 23649 consensus_queue.cc:237] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71a4ef568084fd1a0cc013c4448536d" member_type: VOTER }
I20240501 08:04:06.666471 23646 sys_catalog.cc:564] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:04:06.670518 23650 sys_catalog.cc:455] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c71a4ef568084fd1a0cc013c4448536d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71a4ef568084fd1a0cc013c4448536d" member_type: VOTER } }
I20240501 08:04:06.670821 23651 sys_catalog.cc:455] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c71a4ef568084fd1a0cc013c4448536d. Latest consensus state: current_term: 1 leader_uuid: "c71a4ef568084fd1a0cc013c4448536d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71a4ef568084fd1a0cc013c4448536d" member_type: VOTER } }
I20240501 08:04:06.671083 23650 sys_catalog.cc:458] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [sys.catalog]: This master's current role is: LEADER
I20240501 08:04:06.671281 23651 sys_catalog.cc:458] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [sys.catalog]: This master's current role is: LEADER
I20240501 08:04:06.674127 23656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:04:06.678905 23656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:04:06.682715 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:04:06.687170 23656 catalog_manager.cc:1347] Generated new cluster ID: 2660856b2cc945fbae95d609c7218eba
I20240501 08:04:06.687445 23656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:04:06.697875 23656 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:04:06.699029 23656 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:04:06.716411 23656 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d: Generated new TSK 0
I20240501 08:04:06.716939 23656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:04:06.748791 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:06.755084 23667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:06.756027 23668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:06.757186 23670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:06.758100 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:06.758888 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:06.759078 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:06.759193 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550646759183 us; error 0 us; skew 500 ppm
I20240501 08:04:06.759649 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:06.761801 21084 webserver.cc:460] Webserver started at http://127.20.151.1:44825/ using document root <none> and password file <none>
I20240501 08:04:06.762198 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:06.762393 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:06.762588 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:06.763811 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-0-root/instance:
uuid: "c938ddbe37a74cee94f5b11c8c69f954"
format_stamp: "Formatted at 2024-05-01 08:04:06 on dist-test-slave-pwkb"
I20240501 08:04:06.768596 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:04:06.771589 23675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:06.772249 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:04:06.772531 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-0-root
uuid: "c938ddbe37a74cee94f5b11c8c69f954"
format_stamp: "Formatted at 2024-05-01 08:04:06 on dist-test-slave-pwkb"
I20240501 08:04:06.772776 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:06.788219 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:06.789188 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:06.790575 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:06.792722 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:06.792887 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:06.793066 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:06.793200 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:06.830941 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:36677
I20240501 08:04:06.831039 23737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:36677 every 8 connection(s)
I20240501 08:04:06.835245 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:06.844034 23742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:06.844816 23743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:06.847170 23745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:06.848001 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:06.848699 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:06.848872 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:06.848992 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550646848982 us; error 0 us; skew 500 ppm
I20240501 08:04:06.849429 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:06.849965 23738 heartbeater.cc:347] Connected to a master server at 127.20.151.62:43005
I20240501 08:04:06.850363 23738 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:06.851230 23738 heartbeater.cc:511] Master 127.20.151.62:43005 requested a full tablet report, sending...
I20240501 08:04:06.852095 21084 webserver.cc:460] Webserver started at http://127.20.151.2:44583/ using document root <none> and password file <none>
I20240501 08:04:06.852629 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:06.852854 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:06.853094 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:06.853369 23611 ts_manager.cc:194] Registered new tserver with Master: c938ddbe37a74cee94f5b11c8c69f954 (127.20.151.1:36677)
I20240501 08:04:06.854215 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-1-root/instance:
uuid: "a9e3d86b09fa440096a7377a4911dc2f"
format_stamp: "Formatted at 2024-05-01 08:04:06 on dist-test-slave-pwkb"
I20240501 08:04:06.855059 23611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58740
I20240501 08:04:06.859041 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240501 08:04:06.862228 23750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:06.862963 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240501 08:04:06.863209 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-1-root
uuid: "a9e3d86b09fa440096a7377a4911dc2f"
format_stamp: "Formatted at 2024-05-01 08:04:06 on dist-test-slave-pwkb"
I20240501 08:04:06.863476 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:06.886547 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:06.887663 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:06.888972 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:06.891177 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:06.891372 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:06.891600 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:06.891741 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:06.927942 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:42703
I20240501 08:04:06.928033 23812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:42703 every 8 connection(s)
I20240501 08:04:06.932274 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:06.939702 23816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:06.940299 23817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:06.942940 23813 heartbeater.cc:347] Connected to a master server at 127.20.151.62:43005
W20240501 08:04:06.943234 23819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:06.943277 23813 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:06.944161 23813 heartbeater.cc:511] Master 127.20.151.62:43005 requested a full tablet report, sending...
I20240501 08:04:06.944320 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:06.945210 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:06.945400 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:06.945561 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550646945544 us; error 0 us; skew 500 ppm
I20240501 08:04:06.946175 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:06.946261 23611 ts_manager.cc:194] Registered new tserver with Master: a9e3d86b09fa440096a7377a4911dc2f (127.20.151.2:42703)
I20240501 08:04:06.947983 23611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58752
I20240501 08:04:06.948822 21084 webserver.cc:460] Webserver started at http://127.20.151.3:35709/ using document root <none> and password file <none>
I20240501 08:04:06.949393 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:06.949632 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:06.949975 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:06.951136 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-2-root/instance:
uuid: "bf4b0992bd274ba587e23cc09e270617"
format_stamp: "Formatted at 2024-05-01 08:04:06 on dist-test-slave-pwkb"
I20240501 08:04:06.955016 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 08:04:06.957831 23824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:06.958501 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:04:06.958751 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-2-root
uuid: "bf4b0992bd274ba587e23cc09e270617"
format_stamp: "Formatted at 2024-05-01 08:04:06 on dist-test-slave-pwkb"
I20240501 08:04:06.958998 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:06.989079 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:06.990135 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:06.991437 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:06.993404 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:06.993587 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:06.993803 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:06.993939 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:07.030488 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:40923
I20240501 08:04:07.030593 23886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:40923 every 8 connection(s)
I20240501 08:04:07.034866 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:07.042708 23890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:07.043632 23891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:07.045930 23893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:07.046779 23887 heartbeater.cc:347] Connected to a master server at 127.20.151.62:43005
I20240501 08:04:07.047084 23887 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:07.047726 23887 heartbeater.cc:511] Master 127.20.151.62:43005 requested a full tablet report, sending...
I20240501 08:04:07.049548 23611 ts_manager.cc:194] Registered new tserver with Master: bf4b0992bd274ba587e23cc09e270617 (127.20.151.3:40923)
I20240501 08:04:07.050779 23611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58768
I20240501 08:04:07.051419 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:07.052294 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:07.052485 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:07.052645 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550647052632 us; error 0 us; skew 500 ppm
I20240501 08:04:07.053102 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:07.055166 21084 webserver.cc:460] Webserver started at http://127.20.151.4:37565/ using document root <none> and password file <none>
I20240501 08:04:07.055590 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:07.055773 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:07.056003 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:07.057052 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-3-root/instance:
uuid: "594ad85af22d43d299a986c0e5257922"
format_stamp: "Formatted at 2024-05-01 08:04:07 on dist-test-slave-pwkb"
I20240501 08:04:07.061136 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240501 08:04:07.064123 23898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:07.064960 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:04:07.065205 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-3-root
uuid: "594ad85af22d43d299a986c0e5257922"
format_stamp: "Formatted at 2024-05-01 08:04:07 on dist-test-slave-pwkb"
I20240501 08:04:07.065449 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550573434241-21084-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:07.094522 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:07.095660 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:07.097100 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:07.099340 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:07.099525 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:07.099740 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:07.099874 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:07.155512 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.4:45979
I20240501 08:04:07.155678 23960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.4:45979 every 8 connection(s)
I20240501 08:04:07.170749 23961 heartbeater.cc:347] Connected to a master server at 127.20.151.62:43005
I20240501 08:04:07.171085 23961 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:07.171763 23961 heartbeater.cc:511] Master 127.20.151.62:43005 requested a full tablet report, sending...
I20240501 08:04:07.173508 23611 ts_manager.cc:194] Registered new tserver with Master: 594ad85af22d43d299a986c0e5257922 (127.20.151.4:45979)
I20240501 08:04:07.174693 21084 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.01556448s
I20240501 08:04:07.175045 23611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58784
I20240501 08:04:07.704604 21084 test_util.cc:268] Using random seed: -296858618
I20240501 08:04:07.724009 23611 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58812:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 08:04:07.762467 23852 tablet_service.cc:1467] Processing CreateTablet for tablet 97c9cc8ad8c24568a3a07477127736bf (DEFAULT_TABLE table=test-workload [id=bc64ef3df09c4b1281467c5bbc2aa4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:04:07.763839 23852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c9cc8ad8c24568a3a07477127736bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:07.764781 23778 tablet_service.cc:1467] Processing CreateTablet for tablet 97c9cc8ad8c24568a3a07477127736bf (DEFAULT_TABLE table=test-workload [id=bc64ef3df09c4b1281467c5bbc2aa4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:04:07.765853 23778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c9cc8ad8c24568a3a07477127736bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:07.772207 23926 tablet_service.cc:1467] Processing CreateTablet for tablet 97c9cc8ad8c24568a3a07477127736bf (DEFAULT_TABLE table=test-workload [id=bc64ef3df09c4b1281467c5bbc2aa4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:04:07.773383 23926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c9cc8ad8c24568a3a07477127736bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:07.779438 23982 tablet_bootstrap.cc:492] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f: Bootstrap starting.
I20240501 08:04:07.782727 23983 tablet_bootstrap.cc:492] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617: Bootstrap starting.
I20240501 08:04:07.785912 23982 tablet_bootstrap.cc:654] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:07.787603 23983 tablet_bootstrap.cc:654] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:07.791121 23985 tablet_bootstrap.cc:492] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922: Bootstrap starting.
I20240501 08:04:07.791321 23982 tablet_bootstrap.cc:492] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f: No bootstrap required, opened a new log
I20240501 08:04:07.791752 23982 ts_tablet_manager.cc:1367] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240501 08:04:07.792522 23983 tablet_bootstrap.cc:492] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617: No bootstrap required, opened a new log
I20240501 08:04:07.793040 23983 ts_tablet_manager.cc:1367] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 08:04:07.793951 23982 raft_consensus.cc:348] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e3d86b09fa440096a7377a4911dc2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42703 } } peers { permanent_uuid: "594ad85af22d43d299a986c0e5257922" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45979 } } peers { permanent_uuid: "bf4b0992bd274ba587e23cc09e270617" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40923 } }
I20240501 08:04:07.794786 23982 raft_consensus.cc:374] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:07.795084 23982 raft_consensus.cc:729] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e3d86b09fa440096a7377a4911dc2f, State: Initialized, Role: FOLLOWER
I20240501 08:04:07.795657 23982 consensus_queue.cc:260] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e3d86b09fa440096a7377a4911dc2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42703 } } peers { permanent_uuid: "594ad85af22d43d299a986c0e5257922" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45979 } } peers { permanent_uuid: "bf4b0992bd274ba587e23cc09e270617" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40923 } }
I20240501 08:04:07.796522 23983 raft_consensus.cc:348] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e3d86b09fa440096a7377a4911dc2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42703 } } peers { permanent_uuid: "594ad85af22d43d299a986c0e5257922" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45979 } } peers { permanent_uuid: "bf4b0992bd274ba587e23cc09e270617" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40923 } }
I20240501 08:04:07.797188 23985 tablet_bootstrap.cc:654] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:07.797480 23983 raft_consensus.cc:374] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:07.797807 23813 heartbeater.cc:503] Master 127.20.151.62:43005 was elected leader, sending a full tablet report...
I20240501 08:04:07.798283 23982 ts_tablet_manager.cc:1398] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20240501 08:04:07.798344 23983 raft_consensus.cc:729] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4b0992bd274ba587e23cc09e270617, State: Initialized, Role: FOLLOWER
I20240501 08:04:07.799345 23983 consensus_queue.cc:260] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e3d86b09fa440096a7377a4911dc2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42703 } } peers { permanent_uuid: "594ad85af22d43d299a986c0e5257922" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45979 } } peers { permanent_uuid: "bf4b0992bd274ba587e23cc09e270617" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40923 } }
I20240501 08:04:07.802450 23983 ts_tablet_manager.cc:1398] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20240501 08:04:07.802732 23985 tablet_bootstrap.cc:492] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922: No bootstrap required, opened a new log
I20240501 08:04:07.803167 23985 ts_tablet_manager.cc:1367] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.003s
I20240501 08:04:07.803352 23887 heartbeater.cc:503] Master 127.20.151.62:43005 was elected leader, sending a full tablet report...
I20240501 08:04:07.805861 23985 raft_consensus.cc:348] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e3d86b09fa440096a7377a4911dc2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42703 } } peers { permanent_uuid: "594ad85af22d43d299a986c0e5257922" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45979 } } peers { permanent_uuid: "bf4b0992bd274ba587e23cc09e270617" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40923 } }
I20240501 08:04:07.806634 23985 raft_consensus.cc:374] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:07.806838 23985 raft_consensus.cc:729] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594ad85af22d43d299a986c0e5257922, State: Initialized, Role: FOLLOWER
I20240501 08:04:07.807544 23985 consensus_queue.cc:260] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e3d86b09fa440096a7377a4911dc2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42703 } } peers { permanent_uuid: "594ad85af22d43d299a986c0e5257922" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45979 } } peers { permanent_uuid: "bf4b0992bd274ba587e23cc09e270617" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40923 } }
I20240501 08:04:07.809626 23961 heartbeater.cc:503] Master 127.20.151.62:43005 was elected leader, sending a full tablet report...
I20240501 08:04:07.810148 23985 ts_tablet_manager.cc:1398] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240501 08:04:07.857544 23738 heartbeater.cc:503] Master 127.20.151.62:43005 was elected leader, sending a full tablet report...
W20240501 08:04:07.910554 23962 tablet.cc:2369] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:04:08.005069 23989 raft_consensus.cc:482] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:04:08.005509 23989 raft_consensus.cc:504] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e3d86b09fa440096a7377a4911dc2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42703 } } peers { permanent_uuid: "594ad85af22d43d299a986c0e5257922" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45979 } } peers { permanent_uuid: "bf4b0992bd274ba587e23cc09e270617" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40923 } }
I20240501 08:04:08.007473 23989 leader_election.cc:288] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9e3d86b09fa440096a7377a4911dc2f (127.20.151.2:42703), 594ad85af22d43d299a986c0e5257922 (127.20.151.4:45979)
I20240501 08:04:08.019430 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c9cc8ad8c24568a3a07477127736bf" candidate_uuid: "bf4b0992bd274ba587e23cc09e270617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e3d86b09fa440096a7377a4911dc2f" is_pre_election: true
I20240501 08:04:08.019549 23936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c9cc8ad8c24568a3a07477127736bf" candidate_uuid: "bf4b0992bd274ba587e23cc09e270617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594ad85af22d43d299a986c0e5257922" is_pre_election: true
I20240501 08:04:08.020148 23788 raft_consensus.cc:2454] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf4b0992bd274ba587e23cc09e270617 in term 0.
I20240501 08:04:08.020236 23936 raft_consensus.cc:2454] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf4b0992bd274ba587e23cc09e270617 in term 0.
I20240501 08:04:08.021210 23825 leader_election.cc:302] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e3d86b09fa440096a7377a4911dc2f, bf4b0992bd274ba587e23cc09e270617; no voters:
I20240501 08:04:08.021878 23989 raft_consensus.cc:2781] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:04:08.022140 23989 raft_consensus.cc:482] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:04:08.022383 23989 raft_consensus.cc:3037] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:08.026943 23989 raft_consensus.cc:504] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e3d86b09fa440096a7377a4911dc2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42703 } } peers { permanent_uuid: "594ad85af22d43d299a986c0e5257922" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45979 } } peers { permanent_uuid: "bf4b0992bd274ba587e23cc09e270617" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40923 } }
I20240501 08:04:08.028298 23989 leader_election.cc:288] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [CANDIDATE]: Term 1 election: Requested vote from peers a9e3d86b09fa440096a7377a4911dc2f (127.20.151.2:42703), 594ad85af22d43d299a986c0e5257922 (127.20.151.4:45979)
I20240501 08:04:08.029153 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c9cc8ad8c24568a3a07477127736bf" candidate_uuid: "bf4b0992bd274ba587e23cc09e270617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e3d86b09fa440096a7377a4911dc2f"
I20240501 08:04:08.029264 23936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c9cc8ad8c24568a3a07477127736bf" candidate_uuid: "bf4b0992bd274ba587e23cc09e270617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594ad85af22d43d299a986c0e5257922"
I20240501 08:04:08.029640 23788 raft_consensus.cc:3037] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:08.029726 23936 raft_consensus.cc:3037] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:08.034220 23936 raft_consensus.cc:2454] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf4b0992bd274ba587e23cc09e270617 in term 1.
I20240501 08:04:08.034581 23788 raft_consensus.cc:2454] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf4b0992bd274ba587e23cc09e270617 in term 1.
I20240501 08:04:08.035143 23828 leader_election.cc:302] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 594ad85af22d43d299a986c0e5257922, bf4b0992bd274ba587e23cc09e270617; no voters:
I20240501 08:04:08.035921 23989 raft_consensus.cc:2781] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:04:08.036917 23989 raft_consensus.cc:686] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 1 LEADER]: Becoming Leader. State: Replica: bf4b0992bd274ba587e23cc09e270617, State: Running, Role: LEADER
I20240501 08:04:08.037499 23989 consensus_queue.cc:237] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e3d86b09fa440096a7377a4911dc2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42703 } } peers { permanent_uuid: "594ad85af22d43d299a986c0e5257922" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45979 } } peers { permanent_uuid: "bf4b0992bd274ba587e23cc09e270617" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40923 } }
I20240501 08:04:08.045953 23610 catalog_manager.cc:5484] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf4b0992bd274ba587e23cc09e270617 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "bf4b0992bd274ba587e23cc09e270617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e3d86b09fa440096a7377a4911dc2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "594ad85af22d43d299a986c0e5257922" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf4b0992bd274ba587e23cc09e270617" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40923 } health_report { overall_health: HEALTHY } } }
I20240501 08:04:08.429401 23989 consensus_queue.cc:1035] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e3d86b09fa440096a7377a4911dc2f" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:04:08.440590 23989 consensus_queue.cc:1035] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "594ad85af22d43d299a986c0e5257922" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:04:08.682751 23664 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:08.684410 23664 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:08.684819 23664 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:10.685767 23664 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:10.687031 23664 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:10.687285 23664 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:12.688140 23664 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:12.689334 23664 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:12.689579 23664 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:14.690637 23664 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:14.692510 23664 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:14.692839 23664 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:16.693756 23664 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:16.694994 23664 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:16.695299 23664 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:18.696157 23664 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:18.697432 23664 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:18.697784 23664 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:19.118206 21084 tablet_server.cc:178] TabletServer@127.20.151.1:0 shutting down...
I20240501 08:04:19.141330 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:19.157547 21084 tablet_server.cc:194] TabletServer@127.20.151.1:0 shutdown complete.
I20240501 08:04:19.167340 21084 tablet_server.cc:178] TabletServer@127.20.151.2:0 shutting down...
I20240501 08:04:19.190184 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:19.190874 21084 tablet_replica.cc:330] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f: stopping tablet replica
I20240501 08:04:19.191483 21084 raft_consensus.cc:2229] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:19.192009 21084 raft_consensus.cc:2258] T 97c9cc8ad8c24568a3a07477127736bf P a9e3d86b09fa440096a7377a4911dc2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:19.210565 21084 tablet_server.cc:194] TabletServer@127.20.151.2:0 shutdown complete.
I20240501 08:04:19.220438 21084 tablet_server.cc:178] TabletServer@127.20.151.3:0 shutting down...
I20240501 08:04:19.239643 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:19.240350 21084 tablet_replica.cc:330] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617: stopping tablet replica
I20240501 08:04:19.240928 21084 raft_consensus.cc:2229] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:04:19.241880 21084 raft_consensus.cc:2258] T 97c9cc8ad8c24568a3a07477127736bf P bf4b0992bd274ba587e23cc09e270617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:19.259925 21084 tablet_server.cc:194] TabletServer@127.20.151.3:0 shutdown complete.
I20240501 08:04:19.269512 21084 tablet_server.cc:178] TabletServer@127.20.151.4:0 shutting down...
I20240501 08:04:19.290376 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:19.290967 21084 tablet_replica.cc:330] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922: stopping tablet replica
I20240501 08:04:19.291471 21084 raft_consensus.cc:2229] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:19.291908 21084 raft_consensus.cc:2258] T 97c9cc8ad8c24568a3a07477127736bf P 594ad85af22d43d299a986c0e5257922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:19.310045 21084 tablet_server.cc:194] TabletServer@127.20.151.4:0 shutdown complete.
I20240501 08:04:19.319499 21084 master.cc:537] Master@127.20.151.62:43005 shutting down...
I20240501 08:04:19.338004 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:04:19.338591 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:19.338862 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P c71a4ef568084fd1a0cc013c4448536d: stopping tablet replica
I20240501 08:04:19.356266 21084 master.cc:552] Master@127.20.151.62:43005 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12852 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240501 08:04:19.390524 21084 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.151.62:46871
I20240501 08:04:19.391618 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:19.397584 24005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:19.398614 24006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:19.399647 24008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:19.400171 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:19.400933 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:19.401113 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:19.401259 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550659401241 us; error 0 us; skew 500 ppm
I20240501 08:04:19.401908 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:19.404222 21084 webserver.cc:460] Webserver started at http://127.20.151.62:35591/ using document root <none> and password file <none>
I20240501 08:04:19.404675 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:19.404839 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:19.405092 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:19.406324 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/master-0-root/instance:
uuid: "b1b4d37274cb40c386435044aba287ad"
format_stamp: "Formatted at 2024-05-01 08:04:19 on dist-test-slave-pwkb"
I20240501 08:04:19.410581 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240501 08:04:19.413851 24013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:19.414592 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:04:19.414840 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/master-0-root
uuid: "b1b4d37274cb40c386435044aba287ad"
format_stamp: "Formatted at 2024-05-01 08:04:19 on dist-test-slave-pwkb"
I20240501 08:04:19.415093 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:19.437179 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:19.438338 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:19.474767 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:46871
I20240501 08:04:19.474859 24064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:46871 every 8 connection(s)
I20240501 08:04:19.478580 24065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:19.488986 24065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad: Bootstrap starting.
I20240501 08:04:19.492980 24065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:19.496819 24065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad: No bootstrap required, opened a new log
I20240501 08:04:19.498677 24065 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b4d37274cb40c386435044aba287ad" member_type: VOTER }
I20240501 08:04:19.499119 24065 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:19.499284 24065 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b4d37274cb40c386435044aba287ad, State: Initialized, Role: FOLLOWER
I20240501 08:04:19.499780 24065 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b4d37274cb40c386435044aba287ad" member_type: VOTER }
I20240501 08:04:19.500229 24065 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:04:19.500411 24065 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:04:19.500634 24065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:19.504662 24065 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b4d37274cb40c386435044aba287ad" member_type: VOTER }
I20240501 08:04:19.505079 24065 leader_election.cc:302] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1b4d37274cb40c386435044aba287ad; no voters:
I20240501 08:04:19.506098 24065 leader_election.cc:288] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:04:19.506462 24068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:04:19.507647 24068 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [term 1 LEADER]: Becoming Leader. State: Replica: b1b4d37274cb40c386435044aba287ad, State: Running, Role: LEADER
I20240501 08:04:19.508404 24068 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b4d37274cb40c386435044aba287ad" member_type: VOTER }
I20240501 08:04:19.508875 24065 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:04:19.510788 24069 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1b4d37274cb40c386435044aba287ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b4d37274cb40c386435044aba287ad" member_type: VOTER } }
I20240501 08:04:19.511211 24069 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [sys.catalog]: This master's current role is: LEADER
I20240501 08:04:19.512049 24070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1b4d37274cb40c386435044aba287ad. Latest consensus state: current_term: 1 leader_uuid: "b1b4d37274cb40c386435044aba287ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b4d37274cb40c386435044aba287ad" member_type: VOTER } }
I20240501 08:04:19.513759 24070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [sys.catalog]: This master's current role is: LEADER
I20240501 08:04:19.515472 24073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:04:19.520301 24073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:04:19.523164 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:04:19.528849 24073 catalog_manager.cc:1347] Generated new cluster ID: 626da126cc0d4da6bec01253359b764f
I20240501 08:04:19.529047 24073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:04:19.544214 24073 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:04:19.545346 24073 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:04:19.558931 24073 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad: Generated new TSK 0
I20240501 08:04:19.559422 24073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:04:19.590306 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:19.596825 24086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:19.597868 24087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:19.598551 24089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:19.599613 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:19.600315 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:19.600488 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:19.600620 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550659600608 us; error 0 us; skew 500 ppm
I20240501 08:04:19.601063 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:19.603323 21084 webserver.cc:460] Webserver started at http://127.20.151.1:41847/ using document root <none> and password file <none>
I20240501 08:04:19.603739 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:19.603884 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:19.604089 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:19.605137 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-0-root/instance:
uuid: "78158ce462224783bfb8e68eacc539b9"
format_stamp: "Formatted at 2024-05-01 08:04:19 on dist-test-slave-pwkb"
I20240501 08:04:19.609308 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240501 08:04:19.612547 24094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:19.613261 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 08:04:19.613523 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-0-root
uuid: "78158ce462224783bfb8e68eacc539b9"
format_stamp: "Formatted at 2024-05-01 08:04:19 on dist-test-slave-pwkb"
I20240501 08:04:19.613828 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:19.628561 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:19.629622 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:19.631021 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:19.633209 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:19.633381 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:19.633600 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:19.633817 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240501 08:04:19.671295 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:37965
I20240501 08:04:19.671384 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:37965 every 8 connection(s)
I20240501 08:04:19.675611 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:19.683387 24161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:19.684487 24162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:19.686439 24164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:19.687955 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:19.688561 24157 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46871
I20240501 08:04:19.688977 24157 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:19.688993 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:19.689389 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:19.689577 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550659689560 us; error 0 us; skew 500 ppm
I20240501 08:04:19.689998 24157 heartbeater.cc:511] Master 127.20.151.62:46871 requested a full tablet report, sending...
I20240501 08:04:19.690371 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:19.692234 24030 ts_manager.cc:194] Registered new tserver with Master: 78158ce462224783bfb8e68eacc539b9 (127.20.151.1:37965)
I20240501 08:04:19.693260 21084 webserver.cc:460] Webserver started at http://127.20.151.2:33913/ using document root <none> and password file <none>
I20240501 08:04:19.693743 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:19.693917 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:19.694165 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:19.694152 24030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48758
I20240501 08:04:19.695545 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-1-root/instance:
uuid: "03d902e011d4435592186f060d8ea990"
format_stamp: "Formatted at 2024-05-01 08:04:19 on dist-test-slave-pwkb"
I20240501 08:04:19.699934 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:04:19.703007 24169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:19.703761 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:04:19.704033 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-1-root
uuid: "03d902e011d4435592186f060d8ea990"
format_stamp: "Formatted at 2024-05-01 08:04:19 on dist-test-slave-pwkb"
I20240501 08:04:19.704298 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:19.713949 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:19.715047 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:19.716439 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:19.718621 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:19.718804 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:19.719048 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:19.719200 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:19.755977 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:43677
I20240501 08:04:19.756073 24231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:43677 every 8 connection(s)
I20240501 08:04:19.760617 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:19.767686 24235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:19.769176 24236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:19.769991 24238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:19.771584 24232 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46871
I20240501 08:04:19.771896 24232 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:19.772343 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:19.772532 24232 heartbeater.cc:511] Master 127.20.151.62:46871 requested a full tablet report, sending...
I20240501 08:04:19.773161 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:19.773429 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:19.773667 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550659773647 us; error 0 us; skew 500 ppm
I20240501 08:04:19.774253 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:19.774485 24030 ts_manager.cc:194] Registered new tserver with Master: 03d902e011d4435592186f060d8ea990 (127.20.151.2:43677)
I20240501 08:04:19.775918 24030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53474
I20240501 08:04:19.776783 21084 webserver.cc:460] Webserver started at http://127.20.151.3:37551/ using document root <none> and password file <none>
I20240501 08:04:19.777279 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:19.777448 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:19.777724 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:19.778827 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-2-root/instance:
uuid: "423675f702d34e898c41eba4fc72869d"
format_stamp: "Formatted at 2024-05-01 08:04:19 on dist-test-slave-pwkb"
I20240501 08:04:19.782871 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 08:04:19.785845 24243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:19.786487 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:04:19.786751 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-2-root
uuid: "423675f702d34e898c41eba4fc72869d"
format_stamp: "Formatted at 2024-05-01 08:04:19 on dist-test-slave-pwkb"
I20240501 08:04:19.786998 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550573434241-21084-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:19.801892 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:19.802929 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:19.804265 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:19.806445 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:19.806653 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:19.806869 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:19.807015 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:19.844210 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:41457
I20240501 08:04:19.844311 24305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:41457 every 8 connection(s)
I20240501 08:04:19.856555 24306 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46871
I20240501 08:04:19.856887 24306 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:19.857529 24306 heartbeater.cc:511] Master 127.20.151.62:46871 requested a full tablet report, sending...
I20240501 08:04:19.859202 24030 ts_manager.cc:194] Registered new tserver with Master: 423675f702d34e898c41eba4fc72869d (127.20.151.3:41457)
I20240501 08:04:19.860311 21084 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013271469s
I20240501 08:04:19.860492 24030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53480
I20240501 08:04:20.696877 24157 heartbeater.cc:503] Master 127.20.151.62:46871 was elected leader, sending a full tablet report...
I20240501 08:04:20.778194 24232 heartbeater.cc:503] Master 127.20.151.62:46871 was elected leader, sending a full tablet report...
I20240501 08:04:20.862828 24306 heartbeater.cc:503] Master 127.20.151.62:46871 was elected leader, sending a full tablet report...
I20240501 08:04:20.903522 21084 test_util.cc:268] Using random seed: -283659702
I20240501 08:04:20.925108 24030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53486:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:04:20.927052 24030 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:04:20.966722 24197 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc9bb370c7148c78d419e10a34dafaa (DEFAULT_TABLE table=test-workload [id=e75509a0670f497ab7f9b805879f5b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:04:20.967082 24122 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc9bb370c7148c78d419e10a34dafaa (DEFAULT_TABLE table=test-workload [id=e75509a0670f497ab7f9b805879f5b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:04:20.967963 24197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc9bb370c7148c78d419e10a34dafaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:20.968245 24122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc9bb370c7148c78d419e10a34dafaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:20.971709 24271 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc9bb370c7148c78d419e10a34dafaa (DEFAULT_TABLE table=test-workload [id=e75509a0670f497ab7f9b805879f5b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:04:20.972612 24271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc9bb370c7148c78d419e10a34dafaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:20.984539 24327 tablet_bootstrap.cc:492] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d: Bootstrap starting.
I20240501 08:04:20.989813 24329 tablet_bootstrap.cc:492] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990: Bootstrap starting.
I20240501 08:04:20.990077 24327 tablet_bootstrap.cc:654] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:20.992012 24328 tablet_bootstrap.cc:492] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9: Bootstrap starting.
I20240501 08:04:20.995585 24327 tablet_bootstrap.cc:492] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d: No bootstrap required, opened a new log
I20240501 08:04:20.995637 24329 tablet_bootstrap.cc:654] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:20.996282 24327 ts_tablet_manager.cc:1367] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20240501 08:04:20.997800 24328 tablet_bootstrap.cc:654] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:20.999168 24327 raft_consensus.cc:348] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d902e011d4435592186f060d8ea990" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43677 } } peers { permanent_uuid: "78158ce462224783bfb8e68eacc539b9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37965 } } peers { permanent_uuid: "423675f702d34e898c41eba4fc72869d" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41457 } }
I20240501 08:04:20.999960 24327 raft_consensus.cc:374] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:21.000289 24327 raft_consensus.cc:729] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 423675f702d34e898c41eba4fc72869d, State: Initialized, Role: FOLLOWER
I20240501 08:04:21.000612 24329 tablet_bootstrap.cc:492] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990: No bootstrap required, opened a new log
I20240501 08:04:21.001266 24329 ts_tablet_manager.cc:1367] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240501 08:04:21.001222 24327 consensus_queue.cc:260] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d902e011d4435592186f060d8ea990" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43677 } } peers { permanent_uuid: "78158ce462224783bfb8e68eacc539b9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37965 } } peers { permanent_uuid: "423675f702d34e898c41eba4fc72869d" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41457 } }
I20240501 08:04:21.003574 24329 raft_consensus.cc:348] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d902e011d4435592186f060d8ea990" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43677 } } peers { permanent_uuid: "78158ce462224783bfb8e68eacc539b9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37965 } } peers { permanent_uuid: "423675f702d34e898c41eba4fc72869d" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41457 } }
I20240501 08:04:21.004060 24327 ts_tablet_manager.cc:1398] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:04:21.004204 24329 raft_consensus.cc:374] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:21.004573 24329 raft_consensus.cc:729] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d902e011d4435592186f060d8ea990, State: Initialized, Role: FOLLOWER
I20240501 08:04:21.003233 24328 tablet_bootstrap.cc:492] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9: No bootstrap required, opened a new log
I20240501 08:04:21.005245 24329 consensus_queue.cc:260] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d902e011d4435592186f060d8ea990" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43677 } } peers { permanent_uuid: "78158ce462224783bfb8e68eacc539b9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37965 } } peers { permanent_uuid: "423675f702d34e898c41eba4fc72869d" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41457 } }
I20240501 08:04:21.005828 24328 ts_tablet_manager.cc:1367] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20240501 08:04:21.008569 24328 raft_consensus.cc:348] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d902e011d4435592186f060d8ea990" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43677 } } peers { permanent_uuid: "78158ce462224783bfb8e68eacc539b9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37965 } } peers { permanent_uuid: "423675f702d34e898c41eba4fc72869d" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41457 } }
I20240501 08:04:21.009306 24328 raft_consensus.cc:374] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:21.009603 24328 raft_consensus.cc:729] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78158ce462224783bfb8e68eacc539b9, State: Initialized, Role: FOLLOWER
I20240501 08:04:21.009759 24329 ts_tablet_manager.cc:1398] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240501 08:04:21.010468 24328 consensus_queue.cc:260] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d902e011d4435592186f060d8ea990" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43677 } } peers { permanent_uuid: "78158ce462224783bfb8e68eacc539b9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37965 } } peers { permanent_uuid: "423675f702d34e898c41eba4fc72869d" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41457 } }
I20240501 08:04:21.014117 24328 ts_tablet_manager.cc:1398] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20240501 08:04:21.096962 24333 raft_consensus.cc:482] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:04:21.097441 24333 raft_consensus.cc:504] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d902e011d4435592186f060d8ea990" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43677 } } peers { permanent_uuid: "78158ce462224783bfb8e68eacc539b9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37965 } } peers { permanent_uuid: "423675f702d34e898c41eba4fc72869d" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41457 } }
I20240501 08:04:21.099396 24333 leader_election.cc:288] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03d902e011d4435592186f060d8ea990 (127.20.151.2:43677), 78158ce462224783bfb8e68eacc539b9 (127.20.151.1:37965)
I20240501 08:04:21.109874 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc9bb370c7148c78d419e10a34dafaa" candidate_uuid: "423675f702d34e898c41eba4fc72869d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d902e011d4435592186f060d8ea990" is_pre_election: true
I20240501 08:04:21.110463 24132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc9bb370c7148c78d419e10a34dafaa" candidate_uuid: "423675f702d34e898c41eba4fc72869d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78158ce462224783bfb8e68eacc539b9" is_pre_election: true
I20240501 08:04:21.110671 24207 raft_consensus.cc:2454] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 423675f702d34e898c41eba4fc72869d in term 0.
I20240501 08:04:21.111032 24132 raft_consensus.cc:2454] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 423675f702d34e898c41eba4fc72869d in term 0.
I20240501 08:04:21.111814 24247 leader_election.cc:302] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d902e011d4435592186f060d8ea990, 423675f702d34e898c41eba4fc72869d; no voters:
I20240501 08:04:21.112519 24333 raft_consensus.cc:2781] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:04:21.112800 24333 raft_consensus.cc:482] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:04:21.113045 24333 raft_consensus.cc:3037] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:21.118345 24333 raft_consensus.cc:504] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d902e011d4435592186f060d8ea990" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43677 } } peers { permanent_uuid: "78158ce462224783bfb8e68eacc539b9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37965 } } peers { permanent_uuid: "423675f702d34e898c41eba4fc72869d" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41457 } }
I20240501 08:04:21.119666 24333 leader_election.cc:288] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [CANDIDATE]: Term 1 election: Requested vote from peers 03d902e011d4435592186f060d8ea990 (127.20.151.2:43677), 78158ce462224783bfb8e68eacc539b9 (127.20.151.1:37965)
I20240501 08:04:21.120412 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc9bb370c7148c78d419e10a34dafaa" candidate_uuid: "423675f702d34e898c41eba4fc72869d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d902e011d4435592186f060d8ea990"
I20240501 08:04:21.120529 24132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc9bb370c7148c78d419e10a34dafaa" candidate_uuid: "423675f702d34e898c41eba4fc72869d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78158ce462224783bfb8e68eacc539b9"
I20240501 08:04:21.120855 24207 raft_consensus.cc:3037] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:21.121011 24132 raft_consensus.cc:3037] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:21.125278 24207 raft_consensus.cc:2454] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 423675f702d34e898c41eba4fc72869d in term 1.
I20240501 08:04:21.125584 24132 raft_consensus.cc:2454] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 423675f702d34e898c41eba4fc72869d in term 1.
I20240501 08:04:21.126168 24247 leader_election.cc:302] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d902e011d4435592186f060d8ea990, 423675f702d34e898c41eba4fc72869d; no voters:
I20240501 08:04:21.126951 24333 raft_consensus.cc:2781] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:04:21.128149 24333 raft_consensus.cc:686] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 1 LEADER]: Becoming Leader. State: Replica: 423675f702d34e898c41eba4fc72869d, State: Running, Role: LEADER
I20240501 08:04:21.128849 24333 consensus_queue.cc:237] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d902e011d4435592186f060d8ea990" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43677 } } peers { permanent_uuid: "78158ce462224783bfb8e68eacc539b9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37965 } } peers { permanent_uuid: "423675f702d34e898c41eba4fc72869d" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41457 } }
I20240501 08:04:21.134580 24030 catalog_manager.cc:5484] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d reported cstate change: term changed from 0 to 1, leader changed from <none> to 423675f702d34e898c41eba4fc72869d (127.20.151.3). New cstate: current_term: 1 leader_uuid: "423675f702d34e898c41eba4fc72869d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d902e011d4435592186f060d8ea990" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78158ce462224783bfb8e68eacc539b9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "423675f702d34e898c41eba4fc72869d" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41457 } health_report { overall_health: HEALTHY } } }
I20240501 08:04:21.520958 24078 placement_policy_util.cc:407] tablet 8bc9bb370c7148c78d419e10a34dafaa: detected majority of replicas (2 of 3) at location L0
I20240501 08:04:21.523078 24083 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:21.524401 24083 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:21.524698 24083 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:21.675190 24340 consensus_queue.cc:1035] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78158ce462224783bfb8e68eacc539b9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:04:21.686940 24340 consensus_queue.cc:1035] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d902e011d4435592186f060d8ea990" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:04:22.528945 24078 placement_policy_util.cc:407] tablet 8bc9bb370c7148c78d419e10a34dafaa: detected majority of replicas (2 of 3) at location L0
I20240501 08:04:23.525518 24083 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:23.526897 24083 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:23.527201 24083 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:23.536324 24078 placement_policy_util.cc:407] tablet 8bc9bb370c7148c78d419e10a34dafaa: detected majority of replicas (2 of 3) at location L0
I20240501 08:04:24.544309 24078 placement_policy_util.cc:407] tablet 8bc9bb370c7148c78d419e10a34dafaa: detected majority of replicas (2 of 3) at location L0
I20240501 08:04:25.528167 24083 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:25.529829 24083 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:25.530108 24083 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:25.551568 24078 placement_policy_util.cc:407] tablet 8bc9bb370c7148c78d419e10a34dafaa: detected majority of replicas (2 of 3) at location L0
I20240501 08:04:26.558784 24078 placement_policy_util.cc:407] tablet 8bc9bb370c7148c78d419e10a34dafaa: detected majority of replicas (2 of 3) at location L0
I20240501 08:04:27.530900 24083 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:27.532207 24083 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:27.532492 24083 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:27.566591 24078 placement_policy_util.cc:407] tablet 8bc9bb370c7148c78d419e10a34dafaa: detected majority of replicas (2 of 3) at location L0
I20240501 08:04:28.574463 24078 placement_policy_util.cc:407] tablet 8bc9bb370c7148c78d419e10a34dafaa: detected majority of replicas (2 of 3) at location L0
I20240501 08:04:29.533288 24083 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:29.534584 24083 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:29.534873 24083 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:29.582752 24078 placement_policy_util.cc:407] tablet 8bc9bb370c7148c78d419e10a34dafaa: detected majority of replicas (2 of 3) at location L0
I20240501 08:04:30.589380 24078 placement_policy_util.cc:407] tablet 8bc9bb370c7148c78d419e10a34dafaa: detected majority of replicas (2 of 3) at location L0
I20240501 08:04:31.535758 24083 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:31.536962 24083 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:31.537253 24083 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:31.595180 24078 placement_policy_util.cc:407] tablet 8bc9bb370c7148c78d419e10a34dafaa: detected majority of replicas (2 of 3) at location L0
I20240501 08:04:32.218410 21084 tablet_server.cc:178] TabletServer@127.20.151.1:0 shutting down...
I20240501 08:04:32.239715 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:32.240365 21084 tablet_replica.cc:330] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9: stopping tablet replica
I20240501 08:04:32.240877 21084 raft_consensus.cc:2229] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:32.241390 21084 raft_consensus.cc:2258] T 8bc9bb370c7148c78d419e10a34dafaa P 78158ce462224783bfb8e68eacc539b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:32.260555 21084 tablet_server.cc:194] TabletServer@127.20.151.1:0 shutdown complete.
I20240501 08:04:32.271289 21084 tablet_server.cc:178] TabletServer@127.20.151.2:0 shutting down...
I20240501 08:04:32.291674 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:32.292307 21084 tablet_replica.cc:330] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990: stopping tablet replica
I20240501 08:04:32.292920 21084 raft_consensus.cc:2229] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:32.293608 21084 raft_consensus.cc:2258] T 8bc9bb370c7148c78d419e10a34dafaa P 03d902e011d4435592186f060d8ea990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:32.312088 21084 tablet_server.cc:194] TabletServer@127.20.151.2:0 shutdown complete.
W20240501 08:04:32.320850 24244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.1:37965: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:04:32.324559 24244 consensus_peers.cc:485] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d -> Peer 78158ce462224783bfb8e68eacc539b9 (127.20.151.1:37965): Couldn't send request to peer 78158ce462224783bfb8e68eacc539b9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:37965: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:04:32.326035 21084 tablet_server.cc:178] TabletServer@127.20.151.3:0 shutting down...
I20240501 08:04:32.346201 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:32.346793 21084 tablet_replica.cc:330] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d: stopping tablet replica
I20240501 08:04:32.347304 21084 raft_consensus.cc:2229] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:04:32.348098 21084 raft_consensus.cc:2258] T 8bc9bb370c7148c78d419e10a34dafaa P 423675f702d34e898c41eba4fc72869d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:32.366389 21084 tablet_server.cc:194] TabletServer@127.20.151.3:0 shutdown complete.
I20240501 08:04:32.376034 21084 master.cc:537] Master@127.20.151.62:46871 shutting down...
I20240501 08:04:32.393555 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:04:32.394217 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:32.394574 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P b1b4d37274cb40c386435044aba287ad: stopping tablet replica
I20240501 08:04:32.412330 21084 master.cc:552] Master@127.20.151.62:46871 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (13054 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:516: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] AutoRebalancerTest.TestMaxMovesPerServer (5 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240501 08:04:32.450644 21084 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.151.62:37687
I20240501 08:04:32.451686 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:32.458184 24356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:32.458894 24355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:32.459568 24358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:32.460480 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:32.461153 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:32.461349 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:32.461470 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550672461447 us; error 0 us; skew 500 ppm
I20240501 08:04:32.461913 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:32.464135 21084 webserver.cc:460] Webserver started at http://127.20.151.62:37343/ using document root <none> and password file <none>
I20240501 08:04:32.464524 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:32.464663 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:32.464864 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:32.465983 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/master-0-root/instance:
uuid: "876b9ca1b9e94989bb677ddf5025219b"
format_stamp: "Formatted at 2024-05-01 08:04:32 on dist-test-slave-pwkb"
I20240501 08:04:32.470033 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:04:32.473035 24363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:32.473773 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:04:32.474017 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/master-0-root
uuid: "876b9ca1b9e94989bb677ddf5025219b"
format_stamp: "Formatted at 2024-05-01 08:04:32 on dist-test-slave-pwkb"
I20240501 08:04:32.474277 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:32.499909 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:32.501021 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:32.536633 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:37687
I20240501 08:04:32.536741 24414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:37687 every 8 connection(s)
I20240501 08:04:32.540354 24415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:32.551187 24415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b: Bootstrap starting.
I20240501 08:04:32.555052 24415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:32.559298 24415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b: No bootstrap required, opened a new log
I20240501 08:04:32.561198 24415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876b9ca1b9e94989bb677ddf5025219b" member_type: VOTER }
I20240501 08:04:32.561661 24415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:32.561882 24415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 876b9ca1b9e94989bb677ddf5025219b, State: Initialized, Role: FOLLOWER
I20240501 08:04:32.562448 24415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876b9ca1b9e94989bb677ddf5025219b" member_type: VOTER }
I20240501 08:04:32.562906 24415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:04:32.563108 24415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:04:32.563346 24415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:32.567358 24415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876b9ca1b9e94989bb677ddf5025219b" member_type: VOTER }
I20240501 08:04:32.567828 24415 leader_election.cc:302] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 876b9ca1b9e94989bb677ddf5025219b; no voters:
I20240501 08:04:32.568827 24415 leader_election.cc:288] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:04:32.569119 24418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:04:32.570430 24418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [term 1 LEADER]: Becoming Leader. State: Replica: 876b9ca1b9e94989bb677ddf5025219b, State: Running, Role: LEADER
I20240501 08:04:32.571072 24418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876b9ca1b9e94989bb677ddf5025219b" member_type: VOTER }
I20240501 08:04:32.571620 24415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:04:32.573966 24419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "876b9ca1b9e94989bb677ddf5025219b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876b9ca1b9e94989bb677ddf5025219b" member_type: VOTER } }
I20240501 08:04:32.574136 24420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 876b9ca1b9e94989bb677ddf5025219b. Latest consensus state: current_term: 1 leader_uuid: "876b9ca1b9e94989bb677ddf5025219b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876b9ca1b9e94989bb677ddf5025219b" member_type: VOTER } }
I20240501 08:04:32.574792 24420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [sys.catalog]: This master's current role is: LEADER
I20240501 08:04:32.575937 24419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [sys.catalog]: This master's current role is: LEADER
I20240501 08:04:32.577666 24424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:04:32.582480 24424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:04:32.586915 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:04:32.590871 24424 catalog_manager.cc:1347] Generated new cluster ID: 927e45d0e0f24167a2a714dec278216e
I20240501 08:04:32.591068 24424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:04:32.606062 24424 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:04:32.607156 24424 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:04:32.618367 24424 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b: Generated new TSK 0
I20240501 08:04:32.618865 24424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:04:32.653935 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:32.660457 24436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:32.661310 24437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:32.662252 24439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:32.663162 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:32.663856 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:32.664034 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:32.664178 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550672664162 us; error 0 us; skew 500 ppm
I20240501 08:04:32.664692 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:32.666895 21084 webserver.cc:460] Webserver started at http://127.20.151.1:33273/ using document root <none> and password file <none>
I20240501 08:04:32.667307 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:32.667474 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:32.667702 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:32.668757 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-0-root/instance:
uuid: "3878d9d3d855489f9304c94a510e3076"
format_stamp: "Formatted at 2024-05-01 08:04:32 on dist-test-slave-pwkb"
I20240501 08:04:32.673198 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240501 08:04:32.676340 24444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:32.676990 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 08:04:32.677242 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-0-root
uuid: "3878d9d3d855489f9304c94a510e3076"
format_stamp: "Formatted at 2024-05-01 08:04:32 on dist-test-slave-pwkb"
I20240501 08:04:32.677507 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:32.693883 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:32.694890 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:32.696229 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:32.698372 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:32.698554 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:32.698771 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:32.698932 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:32.735697 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:33159
I20240501 08:04:32.735790 24506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:33159 every 8 connection(s)
I20240501 08:04:32.740033 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:32.748577 24511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:32.749418 24512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:32.750367 24514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:32.752045 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:32.752429 24507 heartbeater.cc:347] Connected to a master server at 127.20.151.62:37687
I20240501 08:04:32.752761 24507 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:32.752848 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:32.753083 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:32.753245 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550672753229 us; error 0 us; skew 500 ppm
I20240501 08:04:32.753494 24507 heartbeater.cc:511] Master 127.20.151.62:37687 requested a full tablet report, sending...
I20240501 08:04:32.753852 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:32.755857 24380 ts_manager.cc:194] Registered new tserver with Master: 3878d9d3d855489f9304c94a510e3076 (127.20.151.1:33159)
I20240501 08:04:32.756603 21084 webserver.cc:460] Webserver started at http://127.20.151.2:40635/ using document root <none> and password file <none>
I20240501 08:04:32.757229 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:32.757499 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:32.757831 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:32.758386 24380 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49388
I20240501 08:04:32.759346 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-1-root/instance:
uuid: "ea8f1c35187941d8ad763c5da2e68fa1"
format_stamp: "Formatted at 2024-05-01 08:04:32 on dist-test-slave-pwkb"
I20240501 08:04:32.763540 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:04:32.766592 24519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:32.767252 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:04:32.767511 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-1-root
uuid: "ea8f1c35187941d8ad763c5da2e68fa1"
format_stamp: "Formatted at 2024-05-01 08:04:32 on dist-test-slave-pwkb"
I20240501 08:04:32.767752 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:32.786055 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:32.787083 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:32.788378 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:32.790441 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:32.790624 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:32.790841 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:32.790990 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:32.827821 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:34865
I20240501 08:04:32.827914 24581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:34865 every 8 connection(s)
I20240501 08:04:32.832487 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:32.840005 24585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:32.840824 24586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:32.843510 24588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:32.844054 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:32.843973 24582 heartbeater.cc:347] Connected to a master server at 127.20.151.62:37687
I20240501 08:04:32.844650 24582 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:32.844969 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:32.845172 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:32.845322 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550672845309 us; error 0 us; skew 500 ppm
I20240501 08:04:32.845355 24582 heartbeater.cc:511] Master 127.20.151.62:37687 requested a full tablet report, sending...
I20240501 08:04:32.845984 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:32.847472 24380 ts_manager.cc:194] Registered new tserver with Master: ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865)
I20240501 08:04:32.848497 21084 webserver.cc:460] Webserver started at http://127.20.151.3:41925/ using document root <none> and password file <none>
I20240501 08:04:32.848892 24380 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49398
I20240501 08:04:32.848955 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:32.849334 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:32.849596 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:32.850823 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-2-root/instance:
uuid: "80abd2e79933401f9ee2d9d4880fc646"
format_stamp: "Formatted at 2024-05-01 08:04:32 on dist-test-slave-pwkb"
I20240501 08:04:32.854892 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:04:32.857822 24593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:32.858503 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:04:32.858764 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-2-root
uuid: "80abd2e79933401f9ee2d9d4880fc646"
format_stamp: "Formatted at 2024-05-01 08:04:32 on dist-test-slave-pwkb"
I20240501 08:04:32.859019 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:32.877964 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:32.879019 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:32.880484 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:32.882709 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:32.882898 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:32.883121 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:32.883271 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:32.919878 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:37287
I20240501 08:04:32.919965 24655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:37287 every 8 connection(s)
I20240501 08:04:32.931986 24656 heartbeater.cc:347] Connected to a master server at 127.20.151.62:37687
I20240501 08:04:32.932317 24656 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:32.932962 24656 heartbeater.cc:511] Master 127.20.151.62:37687 requested a full tablet report, sending...
I20240501 08:04:32.934649 24380 ts_manager.cc:194] Registered new tserver with Master: 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3:37287)
I20240501 08:04:32.935262 21084 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012452641s
I20240501 08:04:32.936254 24380 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49404
I20240501 08:04:33.760824 24507 heartbeater.cc:503] Master 127.20.151.62:37687 was elected leader, sending a full tablet report...
I20240501 08:04:33.851178 24582 heartbeater.cc:503] Master 127.20.151.62:37687 was elected leader, sending a full tablet report...
I20240501 08:04:33.939039 24656 heartbeater.cc:503] Master 127.20.151.62:37687 was elected leader, sending a full tablet report...
I20240501 08:04:33.967778 21084 test_util.cc:268] Using random seed: -270595509
I20240501 08:04:33.996585 24380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49406:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:04:33.999101 24380 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:04:34.054132 24472 tablet_service.cc:1467] Processing CreateTablet for tablet 06132b309aa34d65a3d2271088b7b883 (DEFAULT_TABLE table=test-workload [id=2364b759d2e24623ba3df7682dfde2d0]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 08:04:34.054347 24471 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc237bb85424c3d968dd18ada46214c (DEFAULT_TABLE table=test-workload [id=2364b759d2e24623ba3df7682dfde2d0]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 08:04:34.054759 24470 tablet_service.cc:1467] Processing CreateTablet for tablet 177f26bc23d44e0aad642cf1be6b376e (DEFAULT_TABLE table=test-workload [id=2364b759d2e24623ba3df7682dfde2d0]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 08:04:34.056146 24472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06132b309aa34d65a3d2271088b7b883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:34.056915 24471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc237bb85424c3d968dd18ada46214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:34.059556 24470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177f26bc23d44e0aad642cf1be6b376e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:34.060155 24621 tablet_service.cc:1467] Processing CreateTablet for tablet 06132b309aa34d65a3d2271088b7b883 (DEFAULT_TABLE table=test-workload [id=2364b759d2e24623ba3df7682dfde2d0]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 08:04:34.061494 24621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06132b309aa34d65a3d2271088b7b883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:34.062980 24620 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc237bb85424c3d968dd18ada46214c (DEFAULT_TABLE table=test-workload [id=2364b759d2e24623ba3df7682dfde2d0]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 08:04:34.068779 24546 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc237bb85424c3d968dd18ada46214c (DEFAULT_TABLE table=test-workload [id=2364b759d2e24623ba3df7682dfde2d0]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 08:04:34.064788 24619 tablet_service.cc:1467] Processing CreateTablet for tablet 177f26bc23d44e0aad642cf1be6b376e (DEFAULT_TABLE table=test-workload [id=2364b759d2e24623ba3df7682dfde2d0]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 08:04:34.066442 24547 tablet_service.cc:1467] Processing CreateTablet for tablet 06132b309aa34d65a3d2271088b7b883 (DEFAULT_TABLE table=test-workload [id=2364b759d2e24623ba3df7682dfde2d0]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 08:04:34.070731 24546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc237bb85424c3d968dd18ada46214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:34.072057 24547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06132b309aa34d65a3d2271088b7b883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:34.074007 24620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc237bb85424c3d968dd18ada46214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:34.076536 24619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177f26bc23d44e0aad642cf1be6b376e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:34.078326 24545 tablet_service.cc:1467] Processing CreateTablet for tablet 177f26bc23d44e0aad642cf1be6b376e (DEFAULT_TABLE table=test-workload [id=2364b759d2e24623ba3df7682dfde2d0]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 08:04:34.079605 24545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177f26bc23d44e0aad642cf1be6b376e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:34.104902 24677 tablet_bootstrap.cc:492] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076: Bootstrap starting.
I20240501 08:04:34.111110 24677 tablet_bootstrap.cc:654] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:34.112515 24678 tablet_bootstrap.cc:492] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1: Bootstrap starting.
I20240501 08:04:34.114043 24679 tablet_bootstrap.cc:492] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646: Bootstrap starting.
I20240501 08:04:34.116938 24677 tablet_bootstrap.cc:492] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076: No bootstrap required, opened a new log
I20240501 08:04:34.117457 24677 ts_tablet_manager.cc:1367] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240501 08:04:34.118929 24679 tablet_bootstrap.cc:654] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:34.119336 24678 tablet_bootstrap.cc:654] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:34.120355 24677 raft_consensus.cc:348] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } }
I20240501 08:04:34.121544 24677 raft_consensus.cc:374] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:34.121920 24677 raft_consensus.cc:729] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3878d9d3d855489f9304c94a510e3076, State: Initialized, Role: FOLLOWER
I20240501 08:04:34.123080 24677 consensus_queue.cc:260] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } }
I20240501 08:04:34.127139 24677 ts_tablet_manager.cc:1398] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076: Time spent starting tablet: real 0.009s user 0.002s sys 0.006s
I20240501 08:04:34.127329 24678 tablet_bootstrap.cc:492] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1: No bootstrap required, opened a new log
I20240501 08:04:34.127828 24678 ts_tablet_manager.cc:1367] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20240501 08:04:34.128159 24679 tablet_bootstrap.cc:492] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646: No bootstrap required, opened a new log
I20240501 08:04:34.128599 24679 ts_tablet_manager.cc:1367] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.010s
I20240501 08:04:34.128719 24677 tablet_bootstrap.cc:492] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076: Bootstrap starting.
I20240501 08:04:34.130618 24678 raft_consensus.cc:348] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.131377 24678 raft_consensus.cc:374] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:34.131309 24679 raft_consensus.cc:348] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.131834 24678 raft_consensus.cc:729] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8f1c35187941d8ad763c5da2e68fa1, State: Initialized, Role: FOLLOWER
I20240501 08:04:34.132190 24679 raft_consensus.cc:374] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:34.132475 24679 raft_consensus.cc:729] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80abd2e79933401f9ee2d9d4880fc646, State: Initialized, Role: FOLLOWER
I20240501 08:04:34.132766 24678 consensus_queue.cc:260] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.133199 24679 consensus_queue.cc:260] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.136168 24677 tablet_bootstrap.cc:654] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:34.141073 24678 ts_tablet_manager.cc:1398] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20240501 08:04:34.142092 24679 ts_tablet_manager.cc:1398] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20240501 08:04:34.142957 24678 tablet_bootstrap.cc:492] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1: Bootstrap starting.
I20240501 08:04:34.143121 24679 tablet_bootstrap.cc:492] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646: Bootstrap starting.
I20240501 08:04:34.146440 24677 tablet_bootstrap.cc:492] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076: No bootstrap required, opened a new log
I20240501 08:04:34.147017 24677 ts_tablet_manager.cc:1367] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20240501 08:04:34.148722 24678 tablet_bootstrap.cc:654] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:34.149453 24679 tablet_bootstrap.cc:654] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:34.149538 24677 raft_consensus.cc:348] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.150379 24677 raft_consensus.cc:374] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:34.150712 24677 raft_consensus.cc:729] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3878d9d3d855489f9304c94a510e3076, State: Initialized, Role: FOLLOWER
I20240501 08:04:34.151444 24677 consensus_queue.cc:260] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.154000 24677 ts_tablet_manager.cc:1398] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:04:34.155149 24677 tablet_bootstrap.cc:492] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076: Bootstrap starting.
I20240501 08:04:34.162289 24677 tablet_bootstrap.cc:654] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:34.176882 24677 tablet_bootstrap.cc:492] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076: No bootstrap required, opened a new log
I20240501 08:04:34.177350 24677 ts_tablet_manager.cc:1367] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.012s
I20240501 08:04:34.178835 24679 tablet_bootstrap.cc:492] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646: No bootstrap required, opened a new log
I20240501 08:04:34.179196 24678 tablet_bootstrap.cc:492] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1: No bootstrap required, opened a new log
I20240501 08:04:34.179292 24679 ts_tablet_manager.cc:1367] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.018s
I20240501 08:04:34.179620 24678 ts_tablet_manager.cc:1367] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.031s
I20240501 08:04:34.179947 24677 raft_consensus.cc:348] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.180735 24677 raft_consensus.cc:374] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:34.181049 24677 raft_consensus.cc:729] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3878d9d3d855489f9304c94a510e3076, State: Initialized, Role: FOLLOWER
I20240501 08:04:34.181975 24678 raft_consensus.cc:348] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.181787 24677 consensus_queue.cc:260] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.182520 24678 raft_consensus.cc:374] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:34.182323 24679 raft_consensus.cc:348] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.182809 24678 raft_consensus.cc:729] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8f1c35187941d8ad763c5da2e68fa1, State: Initialized, Role: FOLLOWER
I20240501 08:04:34.183045 24679 raft_consensus.cc:374] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:34.183390 24679 raft_consensus.cc:729] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80abd2e79933401f9ee2d9d4880fc646, State: Initialized, Role: FOLLOWER
I20240501 08:04:34.183629 24678 consensus_queue.cc:260] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.184069 24677 ts_tablet_manager.cc:1398] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20240501 08:04:34.184110 24679 consensus_queue.cc:260] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.186311 24679 ts_tablet_manager.cc:1398] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:04:34.187177 24679 tablet_bootstrap.cc:492] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646: Bootstrap starting.
I20240501 08:04:34.189087 24678 ts_tablet_manager.cc:1398] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240501 08:04:34.189985 24678 tablet_bootstrap.cc:492] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1: Bootstrap starting.
I20240501 08:04:34.192822 24679 tablet_bootstrap.cc:654] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:34.195427 24678 tablet_bootstrap.cc:654] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:34.201098 24679 tablet_bootstrap.cc:492] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646: No bootstrap required, opened a new log
I20240501 08:04:34.201609 24679 ts_tablet_manager.cc:1367] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20240501 08:04:34.203948 24678 tablet_bootstrap.cc:492] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1: No bootstrap required, opened a new log
I20240501 08:04:34.203768 24679 raft_consensus.cc:348] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } }
I20240501 08:04:34.204319 24678 ts_tablet_manager.cc:1367] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
I20240501 08:04:34.204424 24679 raft_consensus.cc:374] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:34.204792 24679 raft_consensus.cc:729] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80abd2e79933401f9ee2d9d4880fc646, State: Initialized, Role: FOLLOWER
I20240501 08:04:34.205605 24685 raft_consensus.cc:482] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:04:34.205376 24679 consensus_queue.cc:260] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } }
I20240501 08:04:34.206099 24685 raft_consensus.cc:504] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.207037 24678 raft_consensus.cc:348] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } }
I20240501 08:04:34.207767 24678 raft_consensus.cc:374] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:34.208042 24678 raft_consensus.cc:729] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8f1c35187941d8ad763c5da2e68fa1, State: Initialized, Role: FOLLOWER
I20240501 08:04:34.208657 24685 leader_election.cc:288] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3878d9d3d855489f9304c94a510e3076 (127.20.151.1:33159), ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865)
I20240501 08:04:34.208845 24678 consensus_queue.cc:260] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } }
I20240501 08:04:34.210104 24684 raft_consensus.cc:482] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:04:34.210539 24684 raft_consensus.cc:504] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.213096 24684 leader_election.cc:288] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3878d9d3d855489f9304c94a510e3076 (127.20.151.1:33159), 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3:37287)
I20240501 08:04:34.221140 24678 ts_tablet_manager.cc:1398] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1: Time spent starting tablet: real 0.017s user 0.001s sys 0.006s
I20240501 08:04:34.228366 24683 raft_consensus.cc:482] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:04:34.229017 24683 raft_consensus.cc:504] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.234115 24683 leader_election.cc:288] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865), 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3:37287)
I20240501 08:04:34.234263 24685 raft_consensus.cc:482] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:04:34.234987 24685 raft_consensus.cc:504] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } }
I20240501 08:04:34.237396 24685 leader_election.cc:288] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865), 3878d9d3d855489f9304c94a510e3076 (127.20.151.1:33159)
I20240501 08:04:34.249869 24679 ts_tablet_manager.cc:1398] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646: Time spent starting tablet: real 0.048s user 0.024s sys 0.023s
I20240501 08:04:34.281198 24482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177f26bc23d44e0aad642cf1be6b376e" candidate_uuid: "80abd2e79933401f9ee2d9d4880fc646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3878d9d3d855489f9304c94a510e3076" is_pre_election: true
I20240501 08:04:34.282196 24482 raft_consensus.cc:2454] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80abd2e79933401f9ee2d9d4880fc646 in term 0.
I20240501 08:04:34.283435 24683 raft_consensus.cc:482] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:04:34.283892 24683 raft_consensus.cc:504] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } }
I20240501 08:04:34.285960 24683 leader_election.cc:288] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3:37287), ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865)
I20240501 08:04:34.287768 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc237bb85424c3d968dd18ada46214c" candidate_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80abd2e79933401f9ee2d9d4880fc646" is_pre_election: true
I20240501 08:04:34.283897 24596 leader_election.cc:302] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3878d9d3d855489f9304c94a510e3076, 80abd2e79933401f9ee2d9d4880fc646; no voters:
I20240501 08:04:34.288630 24631 raft_consensus.cc:2454] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea8f1c35187941d8ad763c5da2e68fa1 in term 0.
I20240501 08:04:34.287825 24482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06132b309aa34d65a3d2271088b7b883" candidate_uuid: "80abd2e79933401f9ee2d9d4880fc646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3878d9d3d855489f9304c94a510e3076" is_pre_election: true
I20240501 08:04:34.289891 24482 raft_consensus.cc:2454] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80abd2e79933401f9ee2d9d4880fc646 in term 0.
I20240501 08:04:34.290103 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc237bb85424c3d968dd18ada46214c" candidate_uuid: "3878d9d3d855489f9304c94a510e3076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80abd2e79933401f9ee2d9d4880fc646" is_pre_election: true
I20240501 08:04:34.290709 24631 raft_consensus.cc:2454] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3878d9d3d855489f9304c94a510e3076 in term 0.
I20240501 08:04:34.290799 24692 raft_consensus.cc:2781] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:04:34.291219 24692 raft_consensus.cc:482] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:04:34.291554 24692 raft_consensus.cc:3037] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:34.291993 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06132b309aa34d65a3d2271088b7b883" candidate_uuid: "3878d9d3d855489f9304c94a510e3076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80abd2e79933401f9ee2d9d4880fc646" is_pre_election: true
I20240501 08:04:34.292325 24596 leader_election.cc:302] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3878d9d3d855489f9304c94a510e3076, 80abd2e79933401f9ee2d9d4880fc646; no voters:
I20240501 08:04:34.292590 24631 raft_consensus.cc:2454] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3878d9d3d855489f9304c94a510e3076 in term 0.
I20240501 08:04:34.293056 24685 raft_consensus.cc:2781] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:04:34.293409 24685 raft_consensus.cc:482] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:04:34.293375 24448 leader_election.cc:302] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3878d9d3d855489f9304c94a510e3076, 80abd2e79933401f9ee2d9d4880fc646; no voters:
I20240501 08:04:34.293964 24685 raft_consensus.cc:3037] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:34.294530 24683 raft_consensus.cc:2781] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:04:34.294731 24448 leader_election.cc:302] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3878d9d3d855489f9304c94a510e3076, 80abd2e79933401f9ee2d9d4880fc646; no voters:
I20240501 08:04:34.294971 24683 raft_consensus.cc:482] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:04:34.296839 24683 raft_consensus.cc:3037] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:34.297096 24557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc237bb85424c3d968dd18ada46214c" candidate_uuid: "3878d9d3d855489f9304c94a510e3076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" is_pre_election: true
I20240501 08:04:34.297830 24557 raft_consensus.cc:2454] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3878d9d3d855489f9304c94a510e3076 in term 0.
I20240501 08:04:34.299276 24557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06132b309aa34d65a3d2271088b7b883" candidate_uuid: "3878d9d3d855489f9304c94a510e3076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" is_pre_election: true
I20240501 08:04:34.299854 24557 raft_consensus.cc:2454] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3878d9d3d855489f9304c94a510e3076 in term 0.
I20240501 08:04:34.301069 24685 raft_consensus.cc:504] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } }
I20240501 08:04:34.305276 24692 raft_consensus.cc:504] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.306933 24685 leader_election.cc:288] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [CANDIDATE]: Term 1 election: Requested vote from peers ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865), 3878d9d3d855489f9304c94a510e3076 (127.20.151.1:33159)
I20240501 08:04:34.306924 24692 leader_election.cc:288] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [CANDIDATE]: Term 1 election: Requested vote from peers 3878d9d3d855489f9304c94a510e3076 (127.20.151.1:33159), ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865)
I20240501 08:04:34.307828 24702 raft_consensus.cc:2781] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:04:34.308162 24702 raft_consensus.cc:482] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:04:34.308393 24702 raft_consensus.cc:3037] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:34.294369 24523 leader_election.cc:302] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80abd2e79933401f9ee2d9d4880fc646, ea8f1c35187941d8ad763c5da2e68fa1; no voters:
I20240501 08:04:34.308805 24683 raft_consensus.cc:504] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.309368 24482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177f26bc23d44e0aad642cf1be6b376e" candidate_uuid: "80abd2e79933401f9ee2d9d4880fc646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3878d9d3d855489f9304c94a510e3076"
I20240501 08:04:34.309556 24684 raft_consensus.cc:2781] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:04:34.309998 24684 raft_consensus.cc:482] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:04:34.310034 24482 raft_consensus.cc:3037] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:34.310492 24684 raft_consensus.cc:3037] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:34.311457 24557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc237bb85424c3d968dd18ada46214c" candidate_uuid: "3878d9d3d855489f9304c94a510e3076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8f1c35187941d8ad763c5da2e68fa1"
I20240501 08:04:34.312419 24683 leader_election.cc:288] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [CANDIDATE]: Term 1 election: Requested vote from peers ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865), 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3:37287)
I20240501 08:04:34.314605 24556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177f26bc23d44e0aad642cf1be6b376e" candidate_uuid: "80abd2e79933401f9ee2d9d4880fc646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" is_pre_election: true
I20240501 08:04:34.314692 24702 raft_consensus.cc:504] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } }
I20240501 08:04:34.315392 24556 raft_consensus.cc:2454] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80abd2e79933401f9ee2d9d4880fc646 in term 0.
I20240501 08:04:34.316725 24702 leader_election.cc:288] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [CANDIDATE]: Term 1 election: Requested vote from peers 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3:37287), ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865)
I20240501 08:04:34.316989 24482 raft_consensus.cc:2454] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80abd2e79933401f9ee2d9d4880fc646 in term 1.
I20240501 08:04:34.317364 24684 raft_consensus.cc:504] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.318120 24555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06132b309aa34d65a3d2271088b7b883" candidate_uuid: "3878d9d3d855489f9304c94a510e3076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8f1c35187941d8ad763c5da2e68fa1"
I20240501 08:04:34.316699 24556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06132b309aa34d65a3d2271088b7b883" candidate_uuid: "80abd2e79933401f9ee2d9d4880fc646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" is_pre_election: true
I20240501 08:04:34.318750 24555 raft_consensus.cc:3037] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:34.319736 24684 leader_election.cc:288] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [CANDIDATE]: Term 1 election: Requested vote from peers 3878d9d3d855489f9304c94a510e3076 (127.20.151.1:33159), 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3:37287)
I20240501 08:04:34.320385 24556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06132b309aa34d65a3d2271088b7b883" candidate_uuid: "80abd2e79933401f9ee2d9d4880fc646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8f1c35187941d8ad763c5da2e68fa1"
I20240501 08:04:34.320385 24596 leader_election.cc:302] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3878d9d3d855489f9304c94a510e3076, 80abd2e79933401f9ee2d9d4880fc646; no voters:
I20240501 08:04:34.321312 24554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177f26bc23d44e0aad642cf1be6b376e" candidate_uuid: "80abd2e79933401f9ee2d9d4880fc646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8f1c35187941d8ad763c5da2e68fa1"
I20240501 08:04:34.322206 24554 raft_consensus.cc:3037] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:34.322209 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc237bb85424c3d968dd18ada46214c" candidate_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80abd2e79933401f9ee2d9d4880fc646"
I20240501 08:04:34.322894 24631 raft_consensus.cc:3037] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:34.324687 24629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06132b309aa34d65a3d2271088b7b883" candidate_uuid: "3878d9d3d855489f9304c94a510e3076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80abd2e79933401f9ee2d9d4880fc646"
I20240501 08:04:34.325572 24629 raft_consensus.cc:2379] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3878d9d3d855489f9304c94a510e3076 in current term 1: Already voted for candidate 80abd2e79933401f9ee2d9d4880fc646 in this term.
I20240501 08:04:34.318732 24557 raft_consensus.cc:2379] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3878d9d3d855489f9304c94a510e3076 in current term 1: Already voted for candidate ea8f1c35187941d8ad763c5da2e68fa1 in this term.
I20240501 08:04:34.325737 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc237bb85424c3d968dd18ada46214c" candidate_uuid: "3878d9d3d855489f9304c94a510e3076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80abd2e79933401f9ee2d9d4880fc646"
I20240501 08:04:34.329874 24554 raft_consensus.cc:2454] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80abd2e79933401f9ee2d9d4880fc646 in term 1.
I20240501 08:04:34.330761 24481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc237bb85424c3d968dd18ada46214c" candidate_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3878d9d3d855489f9304c94a510e3076" is_pre_election: true
I20240501 08:04:34.331000 24447 leader_election.cc:302] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3878d9d3d855489f9304c94a510e3076; no voters: 80abd2e79933401f9ee2d9d4880fc646, ea8f1c35187941d8ad763c5da2e68fa1
I20240501 08:04:34.331632 24481 raft_consensus.cc:2379] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea8f1c35187941d8ad763c5da2e68fa1 in current term 1: Already voted for candidate 3878d9d3d855489f9304c94a510e3076 in this term.
I20240501 08:04:34.332221 24702 raft_consensus.cc:2726] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:04:34.332768 24480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc237bb85424c3d968dd18ada46214c" candidate_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3878d9d3d855489f9304c94a510e3076"
I20240501 08:04:34.333554 24480 raft_consensus.cc:2379] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ea8f1c35187941d8ad763c5da2e68fa1 in current term 1: Already voted for candidate 3878d9d3d855489f9304c94a510e3076 in this term.
I20240501 08:04:34.333961 24555 raft_consensus.cc:2454] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3878d9d3d855489f9304c94a510e3076 in term 1.
I20240501 08:04:34.334661 24631 raft_consensus.cc:2454] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea8f1c35187941d8ad763c5da2e68fa1 in term 1.
I20240501 08:04:34.333989 24692 raft_consensus.cc:2781] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:04:34.335608 24447 leader_election.cc:302] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3878d9d3d855489f9304c94a510e3076, ea8f1c35187941d8ad763c5da2e68fa1; no voters: 80abd2e79933401f9ee2d9d4880fc646
I20240501 08:04:34.335798 24523 leader_election.cc:302] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80abd2e79933401f9ee2d9d4880fc646, ea8f1c35187941d8ad763c5da2e68fa1; no voters: 3878d9d3d855489f9304c94a510e3076
I20240501 08:04:34.336004 24692 raft_consensus.cc:686] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 1 LEADER]: Becoming Leader. State: Replica: 80abd2e79933401f9ee2d9d4880fc646, State: Running, Role: LEADER
I20240501 08:04:34.336608 24684 raft_consensus.cc:2781] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:04:34.336866 24702 raft_consensus.cc:2781] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:04:34.337078 24482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06132b309aa34d65a3d2271088b7b883" candidate_uuid: "80abd2e79933401f9ee2d9d4880fc646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3878d9d3d855489f9304c94a510e3076"
I20240501 08:04:34.337424 24702 raft_consensus.cc:686] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 1 LEADER]: Becoming Leader. State: Replica: 3878d9d3d855489f9304c94a510e3076, State: Running, Role: LEADER
I20240501 08:04:34.338148 24684 raft_consensus.cc:686] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 LEADER]: Becoming Leader. State: Replica: ea8f1c35187941d8ad763c5da2e68fa1, State: Running, Role: LEADER
I20240501 08:04:34.338030 24692 consensus_queue.cc:237] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.339294 24684 consensus_queue.cc:237] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.339294 24702 consensus_queue.cc:237] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } }
I20240501 08:04:34.351208 24596 leader_election.cc:302] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80abd2e79933401f9ee2d9d4880fc646; no voters: 3878d9d3d855489f9304c94a510e3076, ea8f1c35187941d8ad763c5da2e68fa1
I20240501 08:04:34.352301 24692 raft_consensus.cc:2726] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:04:34.351892 24379 catalog_manager.cc:5484] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "80abd2e79933401f9ee2d9d4880fc646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } health_report { overall_health: HEALTHY } } }
I20240501 08:04:34.353101 24380 catalog_manager.cc:5484] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2). New cstate: current_term: 1 leader_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } health_report { overall_health: UNKNOWN } } }
I20240501 08:04:34.353722 24378 catalog_manager.cc:5484] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3878d9d3d855489f9304c94a510e3076 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "3878d9d3d855489f9304c94a510e3076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } health_report { overall_health: HEALTHY } } }
I20240501 08:04:34.435694 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:34.443233 24707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:34.444015 24708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:34.444931 24710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:34.445791 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:34.447443 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:34.447669 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:34.447841 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550674447818 us; error 0 us; skew 500 ppm
I20240501 08:04:34.448390 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:34.451020 21084 webserver.cc:460] Webserver started at http://127.20.151.4:42909/ using document root <none> and password file <none>
I20240501 08:04:34.451506 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:34.451711 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:34.451989 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:34.453181 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-3-root/instance:
uuid: "6b0a3f7737694472bb9888c4890d3093"
format_stamp: "Formatted at 2024-05-01 08:04:34 on dist-test-slave-pwkb"
I20240501 08:04:34.457746 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 08:04:34.460889 24715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:34.461652 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:04:34.461926 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-3-root
uuid: "6b0a3f7737694472bb9888c4890d3093"
format_stamp: "Formatted at 2024-05-01 08:04:34 on dist-test-slave-pwkb"
I20240501 08:04:34.462185 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:34.476704 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:34.477974 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:34.479393 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:34.481943 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:34.482183 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:34.482466 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:34.482707 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:34.520607 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.4:45407
I20240501 08:04:34.520689 24777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.4:45407 every 8 connection(s)
I20240501 08:04:34.534220 24778 heartbeater.cc:347] Connected to a master server at 127.20.151.62:37687
I20240501 08:04:34.534540 24778 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:34.535174 24778 heartbeater.cc:511] Master 127.20.151.62:37687 requested a full tablet report, sending...
I20240501 08:04:34.536839 24378 ts_manager.cc:194] Registered new tserver with Master: 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4:45407)
I20240501 08:04:34.538257 24378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49420
I20240501 08:04:34.586751 24433 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:34.589305 24433 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:34.589670 24433 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 08:04:34.600232 24428 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240501 08:04:34.603433 24428 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9dc237bb85424c3d968dd18ada46214c, TS 80abd2e79933401f9ee2d9d4880fc646 -> TS 6b0a3f7737694472bb9888c4890d3093 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
W20240501 08:04:34.603667 24428 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9dc237bb85424c3d968dd18ada46214c, TS 80abd2e79933401f9ee2d9d4880fc646 -> TS 6b0a3f7737694472bb9888c4890d3093 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
W20240501 08:04:34.607146 24428 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 177f26bc23d44e0aad642cf1be6b376e, TS 3878d9d3d855489f9304c94a510e3076 -> TS 6b0a3f7737694472bb9888c4890d3093 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
W20240501 08:04:34.607348 24428 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 177f26bc23d44e0aad642cf1be6b376e, TS 3878d9d3d855489f9304c94a510e3076 -> TS 6b0a3f7737694472bb9888c4890d3093 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } }
I20240501 08:04:34.610184 24692 consensus_queue.cc:1035] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:04:34.615990 24685 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240501 08:04:34.616223 24685 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240501 08:04:34.643661 24684 consensus_queue.cc:1035] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:04:34.667758 24702 consensus_queue.cc:1035] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:04:34.679661 24788 consensus_queue.cc:1035] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:04:34.705879 24684 consensus_queue.cc:1035] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:04:34.716980 24692 consensus_queue.cc:1035] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:04:35.540345 24778 heartbeater.cc:503] Master 127.20.151.62:37687 was elected leader, sending a full tablet report...
I20240501 08:04:35.616878 24555 consensus_queue.cc:237] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } }
I20240501 08:04:35.622581 24631 raft_consensus.cc:1261] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Refusing update from remote peer ea8f1c35187941d8ad763c5da2e68fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:04:35.622581 24482 raft_consensus.cc:1261] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Refusing update from remote peer ea8f1c35187941d8ad763c5da2e68fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:04:35.624159 24793 consensus_queue.cc:1035] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:04:35.625072 24795 consensus_queue.cc:1035] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:04:35.633265 24482 raft_consensus.cc:2932] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } } }
I20240501 08:04:35.633266 24631 raft_consensus.cc:2932] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } } }
W20240501 08:04:35.635959 24520 consensus_peers.cc:485] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 -> Peer 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4:45407): Couldn't send request to peer 6b0a3f7737694472bb9888c4890d3093. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9dc237bb85424c3d968dd18ada46214c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:04:35.644526 24378 catalog_manager.cc:5484] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 reported cstate change: config changed from index -1 to 2, NON_VOTER 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) added. New cstate: current_term: 1 leader_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } } }
W20240501 08:04:35.820816 24503 debug-util.cc:398] Leaking SignalData structure 0x7b080016faa0 after lost signal to thread 21087
W20240501 08:04:35.821645 24503 debug-util.cc:398] Leaking SignalData structure 0x7b08001ec100 after lost signal to thread 24414
W20240501 08:04:35.822542 24503 debug-util.cc:398] Leaking SignalData structure 0x7b0800047be0 after lost signal to thread 24506
W20240501 08:04:35.823323 24503 debug-util.cc:398] Leaking SignalData structure 0x7b08001b3500 after lost signal to thread 24581
W20240501 08:04:35.824098 24503 debug-util.cc:398] Leaking SignalData structure 0x7b08001a5320 after lost signal to thread 24655
I20240501 08:04:35.894609 24795 raft_consensus.cc:2932] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } } }
I20240501 08:04:35.923139 24631 consensus_queue.cc:237] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } }
I20240501 08:04:35.930568 24555 raft_consensus.cc:1261] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Refusing update from remote peer 80abd2e79933401f9ee2d9d4880fc646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:04:35.933044 24805 consensus_queue.cc:1035] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 08:04:35.936650 24482 raft_consensus.cc:1261] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Refusing update from remote peer 80abd2e79933401f9ee2d9d4880fc646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:04:35.937958 24794 consensus_queue.cc:1035] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:04:35.952713 24594 consensus_peers.cc:485] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 -> Peer 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4:45407): Couldn't send request to peer 6b0a3f7737694472bb9888c4890d3093. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 177f26bc23d44e0aad642cf1be6b376e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:04:35.961401 24480 raft_consensus.cc:2932] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } } }
I20240501 08:04:35.962550 24809 ts_tablet_manager.cc:902] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093: Initiating tablet copy from peer ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865)
I20240501 08:04:35.966064 24555 raft_consensus.cc:2932] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } } }
I20240501 08:04:35.977115 24380 catalog_manager.cc:5484] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 reported cstate change: config changed from index -1 to 2, NON_VOTER 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) added. New cstate: current_term: 1 leader_uuid: "80abd2e79933401f9ee2d9d4880fc646" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } } }
I20240501 08:04:35.984798 24809 tablet_copy_client.cc:308] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.2:34865
I20240501 08:04:35.997972 24567 tablet_copy_service.cc:140] P ea8f1c35187941d8ad763c5da2e68fa1: Received BeginTabletCopySession request for tablet 9dc237bb85424c3d968dd18ada46214c from peer 6b0a3f7737694472bb9888c4890d3093 ({username='slave'} at 127.0.0.1:36138)
I20240501 08:04:35.998456 24567 tablet_copy_service.cc:161] P ea8f1c35187941d8ad763c5da2e68fa1: Beginning new tablet copy session on tablet 9dc237bb85424c3d968dd18ada46214c from peer 6b0a3f7737694472bb9888c4890d3093 at {username='slave'} at 127.0.0.1:36138: session id = 6b0a3f7737694472bb9888c4890d3093-9dc237bb85424c3d968dd18ada46214c
I20240501 08:04:36.004041 24567 tablet_copy_source_session.cc:205] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:04:36.006395 24809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc237bb85424c3d968dd18ada46214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:36.015869 24809 tablet_copy_client.cc:783] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093: tablet copy: Starting download of 0 data blocks...
I20240501 08:04:36.016357 24809 tablet_copy_client.cc:647] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093: tablet copy: Starting download of 1 WAL segments...
I20240501 08:04:36.019981 24809 tablet_copy_client.cc:523] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:04:36.025425 24809 tablet_bootstrap.cc:492] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093: Bootstrap starting.
I20240501 08:04:36.039036 24809 tablet_bootstrap.cc:492] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:04:36.039923 24809 tablet_bootstrap.cc:492] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093: Bootstrap complete.
I20240501 08:04:36.040593 24809 ts_tablet_manager.cc:1367] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.003s
I20240501 08:04:36.043205 24809 raft_consensus.cc:348] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } }
I20240501 08:04:36.044014 24809 raft_consensus.cc:729] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b0a3f7737694472bb9888c4890d3093, State: Initialized, Role: LEARNER
I20240501 08:04:36.044576 24809 consensus_queue.cc:260] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } }
I20240501 08:04:36.047225 24809 ts_tablet_manager.cc:1398] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093: Time spent starting tablet: real 0.006s user 0.005s sys 0.004s
I20240501 08:04:36.048774 24567 tablet_copy_service.cc:342] P ea8f1c35187941d8ad763c5da2e68fa1: Request end of tablet copy session 6b0a3f7737694472bb9888c4890d3093-9dc237bb85424c3d968dd18ada46214c received from {username='slave'} at 127.0.0.1:36138
I20240501 08:04:36.049187 24567 tablet_copy_service.cc:433] P ea8f1c35187941d8ad763c5da2e68fa1: ending tablet copy session 6b0a3f7737694472bb9888c4890d3093-9dc237bb85424c3d968dd18ada46214c on tablet 9dc237bb85424c3d968dd18ada46214c with peer 6b0a3f7737694472bb9888c4890d3093
I20240501 08:04:36.122737 24794 raft_consensus.cc:2932] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } } }
I20240501 08:04:36.232244 24809 ts_tablet_manager.cc:902] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093: Initiating tablet copy from peer 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3:37287)
I20240501 08:04:36.238595 24809 tablet_copy_client.cc:308] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.3:37287
I20240501 08:04:36.248257 24641 tablet_copy_service.cc:140] P 80abd2e79933401f9ee2d9d4880fc646: Received BeginTabletCopySession request for tablet 177f26bc23d44e0aad642cf1be6b376e from peer 6b0a3f7737694472bb9888c4890d3093 ({username='slave'} at 127.0.0.1:41852)
I20240501 08:04:36.248917 24641 tablet_copy_service.cc:161] P 80abd2e79933401f9ee2d9d4880fc646: Beginning new tablet copy session on tablet 177f26bc23d44e0aad642cf1be6b376e from peer 6b0a3f7737694472bb9888c4890d3093 at {username='slave'} at 127.0.0.1:41852: session id = 6b0a3f7737694472bb9888c4890d3093-177f26bc23d44e0aad642cf1be6b376e
I20240501 08:04:36.257414 24641 tablet_copy_source_session.cc:205] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:04:36.260511 24809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177f26bc23d44e0aad642cf1be6b376e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:36.273007 24809 tablet_copy_client.cc:783] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093: tablet copy: Starting download of 0 data blocks...
I20240501 08:04:36.273568 24809 tablet_copy_client.cc:647] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093: tablet copy: Starting download of 1 WAL segments...
I20240501 08:04:36.277662 24809 tablet_copy_client.cc:523] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:04:36.289139 24809 tablet_bootstrap.cc:492] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093: Bootstrap starting.
I20240501 08:04:36.305711 24809 tablet_bootstrap.cc:492] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:04:36.306476 24809 tablet_bootstrap.cc:492] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093: Bootstrap complete.
I20240501 08:04:36.307065 24809 ts_tablet_manager.cc:1367] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093: Time spent bootstrapping tablet: real 0.018s user 0.019s sys 0.001s
I20240501 08:04:36.309342 24809 raft_consensus.cc:348] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } }
I20240501 08:04:36.309954 24809 raft_consensus.cc:729] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b0a3f7737694472bb9888c4890d3093, State: Initialized, Role: LEARNER
I20240501 08:04:36.310608 24809 consensus_queue.cc:260] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: true } }
I20240501 08:04:36.312810 24809 ts_tablet_manager.cc:1398] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 08:04:36.314600 24641 tablet_copy_service.cc:342] P 80abd2e79933401f9ee2d9d4880fc646: Request end of tablet copy session 6b0a3f7737694472bb9888c4890d3093-177f26bc23d44e0aad642cf1be6b376e received from {username='slave'} at 127.0.0.1:41852
I20240501 08:04:36.315032 24641 tablet_copy_service.cc:433] P 80abd2e79933401f9ee2d9d4880fc646: ending tablet copy session 6b0a3f7737694472bb9888c4890d3093-177f26bc23d44e0aad642cf1be6b376e on tablet 177f26bc23d44e0aad642cf1be6b376e with peer 6b0a3f7737694472bb9888c4890d3093
I20240501 08:04:36.323995 24752 raft_consensus.cc:1203] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:04:36.566402 24752 raft_consensus.cc:1203] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:04:36.590564 24433 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:36.594116 24433 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:36.594647 24433 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:36.686302 24703 raft_consensus.cc:1050] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1: attempting to promote NON_VOTER 6b0a3f7737694472bb9888c4890d3093 to VOTER
I20240501 08:04:36.688789 24703 consensus_queue.cc:237] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } }
I20240501 08:04:36.694480 24752 raft_consensus.cc:1261] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [term 1 LEARNER]: Refusing update from remote peer ea8f1c35187941d8ad763c5da2e68fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:04:36.697185 24795 consensus_queue.cc:1035] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 08:04:36.698248 24631 raft_consensus.cc:1261] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Refusing update from remote peer ea8f1c35187941d8ad763c5da2e68fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:04:36.699842 24480 raft_consensus.cc:1261] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Refusing update from remote peer ea8f1c35187941d8ad763c5da2e68fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:04:36.715493 24825 consensus_queue.cc:1035] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 08:04:36.719731 24795 consensus_queue.cc:1035] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:04:36.744956 24752 raft_consensus.cc:2932] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:36.751219 24630 raft_consensus.cc:2932] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:36.759255 24378 catalog_manager.cc:5484] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 reported cstate change: config changed from index 2 to 3, 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:36.762589 24824 raft_consensus.cc:2932] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:36.774740 24480 raft_consensus.cc:2932] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } attrs { replace: true } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:36.810883 24557 consensus_queue.cc:237] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } }
I20240501 08:04:36.816310 24752 raft_consensus.cc:1261] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Refusing update from remote peer ea8f1c35187941d8ad763c5da2e68fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:04:36.817414 24795 consensus_queue.cc:1035] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:04:36.817992 24480 raft_consensus.cc:1261] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Refusing update from remote peer ea8f1c35187941d8ad763c5da2e68fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:04:36.819432 24795 consensus_queue.cc:1035] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:04:36.825551 24752 raft_consensus.cc:2932] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:36.828035 24480 raft_consensus.cc:2932] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:36.840384 24380 catalog_manager.cc:5484] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 reported cstate change: config changed from index 3 to 4, VOTER 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3) evicted. New cstate: current_term: 1 leader_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:36.987272 24619 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dc237bb85424c3d968dd18ada46214c with delete_type TABLET_DATA_TOMBSTONED (TS 80abd2e79933401f9ee2d9d4880fc646 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41798
I20240501 08:04:36.989058 24832 tablet_replica.cc:330] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646: stopping tablet replica
I20240501 08:04:36.990103 24832 raft_consensus.cc:2229] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:36.991176 24832 raft_consensus.cc:2258] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:36.994045 24832 ts_tablet_manager.cc:1875] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:04:37.000763 24794 raft_consensus.cc:1050] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646: attempting to promote NON_VOTER 6b0a3f7737694472bb9888c4890d3093 to VOTER
I20240501 08:04:37.003245 24794 consensus_queue.cc:237] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } }
I20240501 08:04:37.007769 24832 ts_tablet_manager.cc:1888] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:04:37.008122 24832 log.cc:1196] T 9dc237bb85424c3d968dd18ada46214c P 80abd2e79933401f9ee2d9d4880fc646: Deleting WAL directory at /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-2-root/wals/9dc237bb85424c3d968dd18ada46214c
I20240501 08:04:37.009485 24367 catalog_manager.cc:4830] TS 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3:37287): tablet 9dc237bb85424c3d968dd18ada46214c (table test-workload [id=2364b759d2e24623ba3df7682dfde2d0]) successfully deleted
I20240501 08:04:37.011718 24752 raft_consensus.cc:1261] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [term 1 LEARNER]: Refusing update from remote peer 80abd2e79933401f9ee2d9d4880fc646: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:04:37.012097 24555 raft_consensus.cc:1261] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Refusing update from remote peer 80abd2e79933401f9ee2d9d4880fc646: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:04:37.012727 24480 raft_consensus.cc:1261] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Refusing update from remote peer 80abd2e79933401f9ee2d9d4880fc646: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:04:37.013489 24805 consensus_queue.cc:1035] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 08:04:37.015250 24833 consensus_queue.cc:1035] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:04:37.016124 24794 consensus_queue.cc:1035] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:04:37.037884 24480 raft_consensus.cc:2932] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:37.040036 24555 raft_consensus.cc:2932] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:37.054261 24379 catalog_manager.cc:5484] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 reported cstate change: config changed from index 2 to 3, 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80abd2e79933401f9ee2d9d4880fc646" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:37.102990 24794 raft_consensus.cc:2932] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:37.111630 24752 raft_consensus.cc:2932] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6b0a3f7737694472bb9888c4890d3093 (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:37.304474 24630 consensus_queue.cc:237] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } }
I20240501 08:04:37.310990 24752 raft_consensus.cc:1261] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Refusing update from remote peer 80abd2e79933401f9ee2d9d4880fc646: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:04:37.312440 24833 consensus_queue.cc:1035] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:04:37.313800 24555 raft_consensus.cc:1261] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Refusing update from remote peer 80abd2e79933401f9ee2d9d4880fc646: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:04:37.315593 24833 consensus_queue.cc:1035] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:04:37.322187 24752 raft_consensus.cc:2932] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3878d9d3d855489f9304c94a510e3076 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:37.323026 24557 raft_consensus.cc:2932] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3878d9d3d855489f9304c94a510e3076 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:37.337203 24379 catalog_manager.cc:5484] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 reported cstate change: config changed from index 3 to 4, VOTER 3878d9d3d855489f9304c94a510e3076 (127.20.151.1) evicted. New cstate: current_term: 1 leader_uuid: "80abd2e79933401f9ee2d9d4880fc646" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:37.375293 24472 tablet_service.cc:1514] Processing DeleteTablet for tablet 177f26bc23d44e0aad642cf1be6b376e with delete_type TABLET_DATA_TOMBSTONED (TS 3878d9d3d855489f9304c94a510e3076 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51400
I20240501 08:04:37.377017 24837 tablet_replica.cc:330] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076: stopping tablet replica
I20240501 08:04:37.378288 24837 raft_consensus.cc:2229] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:37.379038 24837 raft_consensus.cc:2258] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:37.381665 24837 ts_tablet_manager.cc:1875] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:04:37.387202 24824 raft_consensus.cc:2932] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:37.396297 24837 ts_tablet_manager.cc:1888] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:04:37.396790 24837 log.cc:1196] T 177f26bc23d44e0aad642cf1be6b376e P 3878d9d3d855489f9304c94a510e3076: Deleting WAL directory at /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714550573434241-21084-0/minicluster-data/ts-0-root/wals/177f26bc23d44e0aad642cf1be6b376e
I20240501 08:04:37.397110 24366 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9dc237bb85424c3d968dd18ada46214c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:04:37.398253 24366 catalog_manager.cc:4830] TS 3878d9d3d855489f9304c94a510e3076 (127.20.151.1:33159): tablet 177f26bc23d44e0aad642cf1be6b376e (table test-workload [id=2364b759d2e24623ba3df7682dfde2d0]) successfully deleted
I20240501 08:04:37.877753 24833 raft_consensus.cc:2932] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3878d9d3d855489f9304c94a510e3076 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } } }
I20240501 08:04:37.885185 24367 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 177f26bc23d44e0aad642cf1be6b376e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:04:38.595799 24433 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:38.599900 24433 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:38.600375 24433 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:40.601482 24433 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:40.603885 24433 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:40.604256 24433 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:42.605139 24433 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:42.607664 24433 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:42.607973 24433 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 08:04:43.261351 24652 debug-util.cc:398] Leaking SignalData structure 0x7b08001250a0 after lost signal to thread 21087
W20240501 08:04:43.262204 24652 debug-util.cc:398] Leaking SignalData structure 0x7b08000d2660 after lost signal to thread 24414
W20240501 08:04:43.263129 24652 debug-util.cc:398] Leaking SignalData structure 0x7b0800107760 after lost signal to thread 24506
W20240501 08:04:43.263903 24652 debug-util.cc:398] Leaking SignalData structure 0x7b080007fe00 after lost signal to thread 24581
W20240501 08:04:43.264711 24652 debug-util.cc:398] Leaking SignalData structure 0x7b080000c420 after lost signal to thread 24655
W20240501 08:04:43.265486 24652 debug-util.cc:398] Leaking SignalData structure 0x7b08001c8380 after lost signal to thread 24777
I20240501 08:04:44.609318 24433 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:44.614045 24433 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:44.614555 24433 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:44.861384 21084 tablet_server.cc:178] TabletServer@127.20.151.1:0 shutting down...
I20240501 08:04:44.900269 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:44.901098 21084 tablet_replica.cc:330] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076: stopping tablet replica
I20240501 08:04:44.901852 21084 raft_consensus.cc:2229] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:44.903255 21084 raft_consensus.cc:2258] T 9dc237bb85424c3d968dd18ada46214c P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:44.905754 21084 tablet_replica.cc:330] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076: stopping tablet replica
I20240501 08:04:44.906307 21084 raft_consensus.cc:2229] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:04:44.907330 21084 raft_consensus.cc:2258] T 06132b309aa34d65a3d2271088b7b883 P 3878d9d3d855489f9304c94a510e3076 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 08:04:44.917230 24522 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:04:44.926299 24522 consensus_peers.cc:485] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 -> Peer 3878d9d3d855489f9304c94a510e3076 (127.20.151.1:33159): Couldn't send request to peer 3878d9d3d855489f9304c94a510e3076. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:33159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:04:44.981423 21084 tablet_server.cc:194] TabletServer@127.20.151.1:0 shutdown complete.
I20240501 08:04:45.007354 21084 tablet_server.cc:178] TabletServer@127.20.151.2:0 shutting down...
I20240501 08:04:45.048898 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240501 08:04:45.050472 24596 consensus_peers.cc:485] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 -> Peer ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865): Couldn't send request to peer ea8f1c35187941d8ad763c5da2e68fa1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:04:45.050720 21084 tablet_replica.cc:330] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1: stopping tablet replica
I20240501 08:04:45.051828 21084 raft_consensus.cc:2229] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:45.052399 21084 raft_consensus.cc:2258] T 06132b309aa34d65a3d2271088b7b883 P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:45.054838 21084 tablet_replica.cc:330] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1: stopping tablet replica
I20240501 08:04:45.055378 21084 raft_consensus.cc:2229] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:04:45.056206 21084 raft_consensus.cc:2258] T 9dc237bb85424c3d968dd18ada46214c P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:45.058429 21084 tablet_replica.cc:330] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1: stopping tablet replica
I20240501 08:04:45.058919 21084 raft_consensus.cc:2229] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:45.059926 21084 raft_consensus.cc:2258] T 177f26bc23d44e0aad642cf1be6b376e P ea8f1c35187941d8ad763c5da2e68fa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:45.075002 21084 tablet_server.cc:194] TabletServer@127.20.151.2:0 shutdown complete.
I20240501 08:04:45.093122 21084 tablet_server.cc:178] TabletServer@127.20.151.3:0 shutting down...
I20240501 08:04:45.129305 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:45.130117 21084 tablet_replica.cc:330] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646: stopping tablet replica
I20240501 08:04:45.130793 21084 raft_consensus.cc:2229] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:45.131283 21084 raft_consensus.cc:2258] T 06132b309aa34d65a3d2271088b7b883 P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:45.133592 21084 tablet_replica.cc:330] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646: stopping tablet replica
I20240501 08:04:45.134105 21084 raft_consensus.cc:2229] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:04:45.135036 21084 raft_consensus.cc:2258] T 177f26bc23d44e0aad642cf1be6b376e P 80abd2e79933401f9ee2d9d4880fc646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:45.688299 24846 raft_consensus.cc:482] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 80abd2e79933401f9ee2d9d4880fc646)
I20240501 08:04:45.688771 24846 raft_consensus.cc:504] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "80abd2e79933401f9ee2d9d4880fc646" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37287 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } }
I20240501 08:04:45.691152 24846 leader_election.cc:288] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865), 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3:37287)
W20240501 08:04:45.697194 24718 leader_election.cc:334] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865): Network error: Client connection negotiation failed: client connection to 127.20.151.2:34865: connect: Connection refused (error 111)
W20240501 08:04:45.698042 24719 leader_election.cc:334] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 80abd2e79933401f9ee2d9d4880fc646 (127.20.151.3:37287): Network error: Client connection negotiation failed: client connection to 127.20.151.3:37287: connect: Connection refused (error 111)
I20240501 08:04:45.698524 24719 leader_election.cc:302] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b0a3f7737694472bb9888c4890d3093; no voters: 80abd2e79933401f9ee2d9d4880fc646, ea8f1c35187941d8ad763c5da2e68fa1
I20240501 08:04:45.699384 24846 raft_consensus.cc:2726] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:04:45.951167 24846 raft_consensus.cc:482] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea8f1c35187941d8ad763c5da2e68fa1)
I20240501 08:04:45.951653 24846 raft_consensus.cc:504] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3878d9d3d855489f9304c94a510e3076" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33159 } } peers { permanent_uuid: "ea8f1c35187941d8ad763c5da2e68fa1" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34865 } } peers { permanent_uuid: "6b0a3f7737694472bb9888c4890d3093" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 45407 } attrs { promote: false } }
I20240501 08:04:45.953382 24846 leader_election.cc:288] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3878d9d3d855489f9304c94a510e3076 (127.20.151.1:33159), ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865)
W20240501 08:04:45.957238 24718 leader_election.cc:334] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3878d9d3d855489f9304c94a510e3076 (127.20.151.1:33159): Network error: Client connection negotiation failed: client connection to 127.20.151.1:33159: connect: Connection refused (error 111)
W20240501 08:04:45.957968 24718 leader_election.cc:334] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ea8f1c35187941d8ad763c5da2e68fa1 (127.20.151.2:34865): Network error: Client connection negotiation failed: client connection to 127.20.151.2:34865: connect: Connection refused (error 111)
I20240501 08:04:45.958377 24718 leader_election.cc:302] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b0a3f7737694472bb9888c4890d3093; no voters: 3878d9d3d855489f9304c94a510e3076, ea8f1c35187941d8ad763c5da2e68fa1
I20240501 08:04:45.959143 24846 raft_consensus.cc:2726] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:04:46.046540 21084 tablet_server.cc:194] TabletServer@127.20.151.3:0 shutdown complete.
I20240501 08:04:46.060068 21084 tablet_server.cc:178] TabletServer@127.20.151.4:0 shutting down...
I20240501 08:04:46.081144 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:46.082177 21084 tablet_replica.cc:330] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093: stopping tablet replica
I20240501 08:04:46.082867 21084 raft_consensus.cc:2229] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:46.083635 21084 raft_consensus.cc:2258] T 177f26bc23d44e0aad642cf1be6b376e P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:46.085624 21084 tablet_replica.cc:330] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093: stopping tablet replica
I20240501 08:04:46.086117 21084 raft_consensus.cc:2229] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:46.086687 21084 raft_consensus.cc:2258] T 9dc237bb85424c3d968dd18ada46214c P 6b0a3f7737694472bb9888c4890d3093 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:46.108804 21084 tablet_server.cc:194] TabletServer@127.20.151.4:0 shutdown complete.
I20240501 08:04:46.120224 21084 master.cc:537] Master@127.20.151.62:37687 shutting down...
I20240501 08:04:46.137503 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:04:46.138121 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:46.138446 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P 876b9ca1b9e94989bb677ddf5025219b: stopping tablet replica
I20240501 08:04:46.156271 21084 master.cc:552] Master@127.20.151.62:37687 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (13747 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20240501 08:04:46.199239 21084 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.151.62:37567
I20240501 08:04:46.200203 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:46.207057 24850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:46.207916 24851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:46.208755 24853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:46.209584 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:46.210474 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:46.210646 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:46.210788 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550686210774 us; error 0 us; skew 500 ppm
I20240501 08:04:46.211261 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:46.219326 21084 webserver.cc:460] Webserver started at http://127.20.151.62:35231/ using document root <none> and password file <none>
I20240501 08:04:46.219784 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:46.219961 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:46.220202 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:46.221383 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/master-0-root/instance:
uuid: "8af2ee35a2574b359c35f382e7408e9e"
format_stamp: "Formatted at 2024-05-01 08:04:46 on dist-test-slave-pwkb"
I20240501 08:04:46.225598 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 08:04:46.228842 24858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:46.229656 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:04:46.229947 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/master-0-root
uuid: "8af2ee35a2574b359c35f382e7408e9e"
format_stamp: "Formatted at 2024-05-01 08:04:46 on dist-test-slave-pwkb"
I20240501 08:04:46.230191 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:46.249790 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:46.251080 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:46.286871 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:37567
I20240501 08:04:46.286937 24909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:37567 every 8 connection(s)
I20240501 08:04:46.290529 24910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:46.301036 24910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e: Bootstrap starting.
I20240501 08:04:46.305007 24910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:46.308661 24910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e: No bootstrap required, opened a new log
I20240501 08:04:46.310453 24910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2ee35a2574b359c35f382e7408e9e" member_type: VOTER }
I20240501 08:04:46.310781 24910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:46.310931 24910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af2ee35a2574b359c35f382e7408e9e, State: Initialized, Role: FOLLOWER
I20240501 08:04:46.311367 24910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2ee35a2574b359c35f382e7408e9e" member_type: VOTER }
I20240501 08:04:46.311767 24910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:04:46.311939 24910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:04:46.312156 24910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:46.316267 24910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2ee35a2574b359c35f382e7408e9e" member_type: VOTER }
I20240501 08:04:46.316735 24910 leader_election.cc:302] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8af2ee35a2574b359c35f382e7408e9e; no voters:
I20240501 08:04:46.318220 24910 leader_election.cc:288] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:04:46.318604 24913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:04:46.321659 24910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:04:46.323706 24913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [term 1 LEADER]: Becoming Leader. State: Replica: 8af2ee35a2574b359c35f382e7408e9e, State: Running, Role: LEADER
I20240501 08:04:46.324404 24913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2ee35a2574b359c35f382e7408e9e" member_type: VOTER }
I20240501 08:04:46.332140 24915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8af2ee35a2574b359c35f382e7408e9e. Latest consensus state: current_term: 1 leader_uuid: "8af2ee35a2574b359c35f382e7408e9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2ee35a2574b359c35f382e7408e9e" member_type: VOTER } }
I20240501 08:04:46.332738 24915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [sys.catalog]: This master's current role is: LEADER
I20240501 08:04:46.333302 24914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8af2ee35a2574b359c35f382e7408e9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2ee35a2574b359c35f382e7408e9e" member_type: VOTER } }
I20240501 08:04:46.334003 24914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [sys.catalog]: This master's current role is: LEADER
I20240501 08:04:46.336266 24922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:04:46.342466 24922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:04:46.345665 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:04:46.350512 24922 catalog_manager.cc:1347] Generated new cluster ID: a7e2fbd1d20a4d658bdce613dde92b2c
I20240501 08:04:46.350723 24922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:04:46.365188 24922 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:04:46.366436 24922 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:04:46.378921 24922 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e: Generated new TSK 0
I20240501 08:04:46.379459 24922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:04:46.411921 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:46.418640 24931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:46.419627 24932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:46.420363 24934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:46.422336 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:46.423210 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:46.423384 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:46.423539 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550686423526 us; error 0 us; skew 500 ppm
I20240501 08:04:46.424082 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:46.426388 21084 webserver.cc:460] Webserver started at http://127.20.151.1:40995/ using document root <none> and password file <none>
I20240501 08:04:46.426828 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:46.426992 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:46.427263 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:46.428308 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-0-root/instance:
uuid: "a8927c3ea2f74614b9b194e164a08602"
format_stamp: "Formatted at 2024-05-01 08:04:46 on dist-test-slave-pwkb"
I20240501 08:04:46.432955 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:04:46.436067 24939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:46.436723 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:04:46.436986 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-0-root
uuid: "a8927c3ea2f74614b9b194e164a08602"
format_stamp: "Formatted at 2024-05-01 08:04:46 on dist-test-slave-pwkb"
I20240501 08:04:46.437244 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:46.468884 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:46.469971 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:46.471350 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:46.473518 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:46.473737 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240501 08:04:46.473955 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:46.474120 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:46.512588 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:39039
I20240501 08:04:46.512691 25001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:39039 every 8 connection(s)
I20240501 08:04:46.518317 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:46.526882 25006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:46.527213 25007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:46.530259 21084 server_base.cc:1034] running on GCE node
W20240501 08:04:46.531013 25009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:46.531483 25002 heartbeater.cc:347] Connected to a master server at 127.20.151.62:37567
I20240501 08:04:46.531884 25002 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:46.532033 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:46.532294 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:46.532488 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550686532471 us; error 0 us; skew 500 ppm
I20240501 08:04:46.532763 25002 heartbeater.cc:511] Master 127.20.151.62:37567 requested a full tablet report, sending...
I20240501 08:04:46.533267 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:46.534768 24875 ts_manager.cc:194] Registered new tserver with Master: a8927c3ea2f74614b9b194e164a08602 (127.20.151.1:39039)
I20240501 08:04:46.536468 24875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37242
I20240501 08:04:46.538525 21084 webserver.cc:460] Webserver started at http://127.20.151.2:45763/ using document root <none> and password file <none>
I20240501 08:04:46.538982 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:46.539191 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:46.539404 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:46.540371 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-1-root/instance:
uuid: "08e89c1dd10f432fb5516b18aa5cce84"
format_stamp: "Formatted at 2024-05-01 08:04:46 on dist-test-slave-pwkb"
I20240501 08:04:46.544569 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:04:46.547751 25014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:46.548601 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:04:46.548853 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-1-root
uuid: "08e89c1dd10f432fb5516b18aa5cce84"
format_stamp: "Formatted at 2024-05-01 08:04:46 on dist-test-slave-pwkb"
I20240501 08:04:46.549095 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:46.569334 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:46.570466 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:46.571776 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:46.573887 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:46.574079 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:46.574298 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:46.574441 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:46.611235 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:35059
I20240501 08:04:46.611323 25076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:35059 every 8 connection(s)
I20240501 08:04:46.615628 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:46.623615 25080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:46.624641 25081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:46.626190 25083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:46.626731 25077 heartbeater.cc:347] Connected to a master server at 127.20.151.62:37567
I20240501 08:04:46.627048 25077 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:46.627486 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:46.627724 25077 heartbeater.cc:511] Master 127.20.151.62:37567 requested a full tablet report, sending...
I20240501 08:04:46.628226 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:46.628454 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:46.628683 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550686628636 us; error 0 us; skew 500 ppm
I20240501 08:04:46.629314 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:46.629611 24875 ts_manager.cc:194] Registered new tserver with Master: 08e89c1dd10f432fb5516b18aa5cce84 (127.20.151.2:35059)
I20240501 08:04:46.631176 24875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37244
I20240501 08:04:46.632176 21084 webserver.cc:460] Webserver started at http://127.20.151.3:33373/ using document root <none> and password file <none>
I20240501 08:04:46.632726 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:46.632894 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:46.633149 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:46.634274 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-2-root/instance:
uuid: "17766bfac44e4187887d8a1dddbd74d2"
format_stamp: "Formatted at 2024-05-01 08:04:46 on dist-test-slave-pwkb"
I20240501 08:04:46.638392 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 08:04:46.641289 25088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:46.642030 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:04:46.642273 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-2-root
uuid: "17766bfac44e4187887d8a1dddbd74d2"
format_stamp: "Formatted at 2024-05-01 08:04:46 on dist-test-slave-pwkb"
I20240501 08:04:46.642505 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714550573434241-21084-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:46.659994 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:46.660959 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:46.662324 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:46.664422 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:46.664608 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:46.664834 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:46.664996 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:46.702077 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:44447
I20240501 08:04:46.702158 25150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:44447 every 8 connection(s)
I20240501 08:04:46.715225 25151 heartbeater.cc:347] Connected to a master server at 127.20.151.62:37567
I20240501 08:04:46.715564 25151 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:46.716243 25151 heartbeater.cc:511] Master 127.20.151.62:37567 requested a full tablet report, sending...
I20240501 08:04:46.718077 24875 ts_manager.cc:194] Registered new tserver with Master: 17766bfac44e4187887d8a1dddbd74d2 (127.20.151.3:44447)
I20240501 08:04:46.718276 21084 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013244959s
I20240501 08:04:46.720003 24875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37246
I20240501 08:04:47.539052 25002 heartbeater.cc:503] Master 127.20.151.62:37567 was elected leader, sending a full tablet report...
I20240501 08:04:47.633806 25077 heartbeater.cc:503] Master 127.20.151.62:37567 was elected leader, sending a full tablet report...
I20240501 08:04:47.722700 25151 heartbeater.cc:503] Master 127.20.151.62:37567 was elected leader, sending a full tablet report...
I20240501 08:04:47.751106 21084 test_util.cc:268] Using random seed: -256812116
I20240501 08:04:47.772550 24875 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37256:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:04:47.774446 24875 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:04:47.820886 24967 tablet_service.cc:1467] Processing CreateTablet for tablet 6318bac57a7c4c6180f8b953946ad8c4 (DEFAULT_TABLE table=test-workload [id=32bcbaae5a254ac296cf8e0384b70e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:04:47.822266 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6318bac57a7c4c6180f8b953946ad8c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:47.829223 25116 tablet_service.cc:1467] Processing CreateTablet for tablet 6318bac57a7c4c6180f8b953946ad8c4 (DEFAULT_TABLE table=test-workload [id=32bcbaae5a254ac296cf8e0384b70e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:04:47.830045 25042 tablet_service.cc:1467] Processing CreateTablet for tablet 6318bac57a7c4c6180f8b953946ad8c4 (DEFAULT_TABLE table=test-workload [id=32bcbaae5a254ac296cf8e0384b70e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:04:47.830754 25116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6318bac57a7c4c6180f8b953946ad8c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:47.831326 25042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6318bac57a7c4c6180f8b953946ad8c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:47.840072 25171 tablet_bootstrap.cc:492] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602: Bootstrap starting.
I20240501 08:04:47.848131 25171 tablet_bootstrap.cc:654] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:47.849606 25172 tablet_bootstrap.cc:492] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84: Bootstrap starting.
I20240501 08:04:47.851385 25173 tablet_bootstrap.cc:492] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2: Bootstrap starting.
I20240501 08:04:47.854194 25171 tablet_bootstrap.cc:492] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602: No bootstrap required, opened a new log
I20240501 08:04:47.854631 25171 ts_tablet_manager.cc:1367] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20240501 08:04:47.856220 25172 tablet_bootstrap.cc:654] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:47.856779 25171 raft_consensus.cc:348] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } }
I20240501 08:04:47.857304 25171 raft_consensus.cc:374] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:47.857542 25171 raft_consensus.cc:729] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8927c3ea2f74614b9b194e164a08602, State: Initialized, Role: FOLLOWER
I20240501 08:04:47.857731 25173 tablet_bootstrap.cc:654] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:47.858136 25171 consensus_queue.cc:260] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } }
I20240501 08:04:47.860805 25171 ts_tablet_manager.cc:1398] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240501 08:04:47.863960 25172 tablet_bootstrap.cc:492] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84: No bootstrap required, opened a new log
I20240501 08:04:47.864434 25172 ts_tablet_manager.cc:1367] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240501 08:04:47.864966 25173 tablet_bootstrap.cc:492] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2: No bootstrap required, opened a new log
I20240501 08:04:47.865921 25173 ts_tablet_manager.cc:1367] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240501 08:04:47.867120 25172 raft_consensus.cc:348] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } }
I20240501 08:04:47.867841 25172 raft_consensus.cc:374] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:47.867908 25173 raft_consensus.cc:348] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } }
I20240501 08:04:47.868196 25172 raft_consensus.cc:729] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e89c1dd10f432fb5516b18aa5cce84, State: Initialized, Role: FOLLOWER
I20240501 08:04:47.868561 25173 raft_consensus.cc:374] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:47.868871 25173 raft_consensus.cc:729] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17766bfac44e4187887d8a1dddbd74d2, State: Initialized, Role: FOLLOWER
I20240501 08:04:47.869122 25172 consensus_queue.cc:260] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } }
I20240501 08:04:47.869414 25173 consensus_queue.cc:260] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } }
I20240501 08:04:47.871973 25172 ts_tablet_manager.cc:1398] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240501 08:04:47.875174 25173 ts_tablet_manager.cc:1398] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2: Time spent starting tablet: real 0.009s user 0.000s sys 0.008s
I20240501 08:04:47.886726 25175 raft_consensus.cc:482] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:04:47.887096 25175 raft_consensus.cc:504] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } }
I20240501 08:04:47.888793 25175 leader_election.cc:288] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17766bfac44e4187887d8a1dddbd74d2 (127.20.151.3:44447), 08e89c1dd10f432fb5516b18aa5cce84 (127.20.151.2:35059)
I20240501 08:04:47.899889 25126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6318bac57a7c4c6180f8b953946ad8c4" candidate_uuid: "a8927c3ea2f74614b9b194e164a08602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17766bfac44e4187887d8a1dddbd74d2" is_pre_election: true
I20240501 08:04:47.900607 25052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6318bac57a7c4c6180f8b953946ad8c4" candidate_uuid: "a8927c3ea2f74614b9b194e164a08602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e89c1dd10f432fb5516b18aa5cce84" is_pre_election: true
I20240501 08:04:47.900712 25126 raft_consensus.cc:2454] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8927c3ea2f74614b9b194e164a08602 in term 0.
I20240501 08:04:47.901364 25052 raft_consensus.cc:2454] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8927c3ea2f74614b9b194e164a08602 in term 0.
I20240501 08:04:47.902056 24943 leader_election.cc:302] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17766bfac44e4187887d8a1dddbd74d2, a8927c3ea2f74614b9b194e164a08602; no voters:
I20240501 08:04:47.902832 25175 raft_consensus.cc:2781] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:04:47.903086 25175 raft_consensus.cc:482] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:04:47.903342 25175 raft_consensus.cc:3037] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:47.907866 25175 raft_consensus.cc:504] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } }
I20240501 08:04:47.909534 25175 leader_election.cc:288] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [CANDIDATE]: Term 1 election: Requested vote from peers 17766bfac44e4187887d8a1dddbd74d2 (127.20.151.3:44447), 08e89c1dd10f432fb5516b18aa5cce84 (127.20.151.2:35059)
I20240501 08:04:47.910174 25126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6318bac57a7c4c6180f8b953946ad8c4" candidate_uuid: "a8927c3ea2f74614b9b194e164a08602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17766bfac44e4187887d8a1dddbd74d2"
I20240501 08:04:47.910419 25052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6318bac57a7c4c6180f8b953946ad8c4" candidate_uuid: "a8927c3ea2f74614b9b194e164a08602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e89c1dd10f432fb5516b18aa5cce84"
I20240501 08:04:47.910600 25126 raft_consensus.cc:3037] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:47.910913 25052 raft_consensus.cc:3037] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:47.915163 25126 raft_consensus.cc:2454] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8927c3ea2f74614b9b194e164a08602 in term 1.
I20240501 08:04:47.915244 25052 raft_consensus.cc:2454] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8927c3ea2f74614b9b194e164a08602 in term 1.
I20240501 08:04:47.916103 24943 leader_election.cc:302] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17766bfac44e4187887d8a1dddbd74d2, a8927c3ea2f74614b9b194e164a08602; no voters:
I20240501 08:04:47.916682 25175 raft_consensus.cc:2781] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:04:47.917732 25175 raft_consensus.cc:686] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 1 LEADER]: Becoming Leader. State: Replica: a8927c3ea2f74614b9b194e164a08602, State: Running, Role: LEADER
I20240501 08:04:47.918705 25175 consensus_queue.cc:237] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } }
I20240501 08:04:47.926286 24875 catalog_manager.cc:5484] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8927c3ea2f74614b9b194e164a08602 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "a8927c3ea2f74614b9b194e164a08602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } health_report { overall_health: UNKNOWN } } }
I20240501 08:04:47.955125 21084 tablet_server.cc:178] TabletServer@127.20.151.1:0 shutting down...
W20240501 08:04:47.957653 25152 tablet.cc:2369] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:04:47.972889 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:47.973680 21084 tablet_replica.cc:330] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602: stopping tablet replica
I20240501 08:04:47.974215 21084 raft_consensus.cc:2229] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:04:47.974756 21084 pending_rounds.cc:62] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602: Trying to abort 1 pending ops.
I20240501 08:04:47.974941 21084 pending_rounds.cc:68] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022799617720897536 op_type: NO_OP noop_request { }
I20240501 08:04:47.975217 21084 raft_consensus.cc:2866] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 08:04:47.975474 21084 raft_consensus.cc:2258] T 6318bac57a7c4c6180f8b953946ad8c4 P a8927c3ea2f74614b9b194e164a08602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:47.994414 21084 tablet_server.cc:194] TabletServer@127.20.151.1:0 shutdown complete.
I20240501 08:04:48.345329 24928 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:48.346587 24928 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:48.346858 24928 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:49.579926 25184 raft_consensus.cc:482] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:04:49.580340 25184 raft_consensus.cc:504] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } }
I20240501 08:04:49.582119 25184 leader_election.cc:288] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8927c3ea2f74614b9b194e164a08602 (127.20.151.1:39039), 17766bfac44e4187887d8a1dddbd74d2 (127.20.151.3:44447)
W20240501 08:04:49.588383 25017 leader_election.cc:334] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a8927c3ea2f74614b9b194e164a08602 (127.20.151.1:39039): Network error: Client connection negotiation failed: client connection to 127.20.151.1:39039: connect: Connection refused (error 111)
I20240501 08:04:49.592924 25126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6318bac57a7c4c6180f8b953946ad8c4" candidate_uuid: "08e89c1dd10f432fb5516b18aa5cce84" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17766bfac44e4187887d8a1dddbd74d2" is_pre_election: true
I20240501 08:04:49.593473 25126 raft_consensus.cc:2454] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08e89c1dd10f432fb5516b18aa5cce84 in term 1.
I20240501 08:04:49.594298 25018 leader_election.cc:302] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08e89c1dd10f432fb5516b18aa5cce84, 17766bfac44e4187887d8a1dddbd74d2; no voters: a8927c3ea2f74614b9b194e164a08602
I20240501 08:04:49.594925 25184 raft_consensus.cc:2781] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:04:49.595244 25184 raft_consensus.cc:482] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:04:49.595461 25184 raft_consensus.cc:3037] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:04:49.599927 25184 raft_consensus.cc:504] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } }
I20240501 08:04:49.601435 25184 leader_election.cc:288] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [CANDIDATE]: Term 2 election: Requested vote from peers a8927c3ea2f74614b9b194e164a08602 (127.20.151.1:39039), 17766bfac44e4187887d8a1dddbd74d2 (127.20.151.3:44447)
I20240501 08:04:49.602483 25126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6318bac57a7c4c6180f8b953946ad8c4" candidate_uuid: "08e89c1dd10f432fb5516b18aa5cce84" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17766bfac44e4187887d8a1dddbd74d2"
I20240501 08:04:49.602958 25126 raft_consensus.cc:3037] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:04:49.604382 25017 leader_election.cc:334] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a8927c3ea2f74614b9b194e164a08602 (127.20.151.1:39039): Network error: Client connection negotiation failed: client connection to 127.20.151.1:39039: connect: Connection refused (error 111)
I20240501 08:04:49.607403 25126 raft_consensus.cc:2454] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08e89c1dd10f432fb5516b18aa5cce84 in term 2.
I20240501 08:04:49.608189 25018 leader_election.cc:302] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08e89c1dd10f432fb5516b18aa5cce84, 17766bfac44e4187887d8a1dddbd74d2; no voters: a8927c3ea2f74614b9b194e164a08602
I20240501 08:04:49.608764 25184 raft_consensus.cc:2781] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:04:49.609736 25184 raft_consensus.cc:686] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 2 LEADER]: Becoming Leader. State: Replica: 08e89c1dd10f432fb5516b18aa5cce84, State: Running, Role: LEADER
I20240501 08:04:49.610337 25184 consensus_queue.cc:237] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } }
I20240501 08:04:49.616225 24874 catalog_manager.cc:5484] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 reported cstate change: term changed from 1 to 2, leader changed from a8927c3ea2f74614b9b194e164a08602 (127.20.151.1) to 08e89c1dd10f432fb5516b18aa5cce84 (127.20.151.2). New cstate: current_term: 2 leader_uuid: "08e89c1dd10f432fb5516b18aa5cce84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8927c3ea2f74614b9b194e164a08602" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 39039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08e89c1dd10f432fb5516b18aa5cce84" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 35059 } health_report { overall_health: HEALTHY } } }
I20240501 08:04:50.129308 25126 raft_consensus.cc:1261] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [term 2 FOLLOWER]: Refusing update from remote peer 08e89c1dd10f432fb5516b18aa5cce84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 08:04:50.130638 25195 consensus_queue.cc:1035] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17766bfac44e4187887d8a1dddbd74d2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:04:50.153102 25017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.1:39039: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240501 08:04:50.156222 25017 consensus_peers.cc:485] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 -> Peer a8927c3ea2f74614b9b194e164a08602 (127.20.151.1:39039): Couldn't send request to peer a8927c3ea2f74614b9b194e164a08602. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:39039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:04:50.347641 24928 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:50.348963 24928 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:50.349254 24928 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:52.350098 24928 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:52.351421 24928 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:52.351790 24928 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 08:04:52.756587 25017 consensus_peers.cc:485] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 -> Peer a8927c3ea2f74614b9b194e164a08602 (127.20.151.1:39039): Couldn't send request to peer a8927c3ea2f74614b9b194e164a08602. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:39039: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:04:54.853842 24928 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:54.855943 24928 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:54.856362 24928 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 08:04:55.362861 25017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.1:39039: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:04:55.365728 25017 consensus_peers.cc:485] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 -> Peer a8927c3ea2f74614b9b194e164a08602 (127.20.151.1:39039): Couldn't send request to peer a8927c3ea2f74614b9b194e164a08602. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:39039: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:04:55.388772 25073 debug-util.cc:398] Leaking SignalData structure 0x7b0800095aa0 after lost signal to thread 21087
W20240501 08:04:55.389658 25073 debug-util.cc:398] Leaking SignalData structure 0x7b08000df420 after lost signal to thread 24909
W20240501 08:04:55.390722 25073 debug-util.cc:398] Leaking SignalData structure 0x7b08000fe560 after lost signal to thread 25076
W20240501 08:04:55.391559 25073 debug-util.cc:398] Leaking SignalData structure 0x7b08001a2160 after lost signal to thread 25150
I20240501 08:04:56.857240 24928 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:56.858551 24928 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:56.858907 24928 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 08:04:57.955782 25017 consensus_peers.cc:485] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 -> Peer a8927c3ea2f74614b9b194e164a08602 (127.20.151.1:39039): Couldn't send request to peer a8927c3ea2f74614b9b194e164a08602. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:39039: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:04:58.859704 24928 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:04:58.861045 24928 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:04:58.861354 24928 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:04:59.227006 21084 tablet_server.cc:178] TabletServer@127.20.151.2:0 shutting down...
I20240501 08:04:59.251298 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:59.251940 21084 tablet_replica.cc:330] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84: stopping tablet replica
I20240501 08:04:59.252521 21084 raft_consensus.cc:2229] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:04:59.253495 21084 raft_consensus.cc:2258] T 6318bac57a7c4c6180f8b953946ad8c4 P 08e89c1dd10f432fb5516b18aa5cce84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:59.288566 21084 tablet_server.cc:194] TabletServer@127.20.151.2:0 shutdown complete.
I20240501 08:04:59.299185 21084 tablet_server.cc:178] TabletServer@127.20.151.3:0 shutting down...
I20240501 08:04:59.320235 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:04:59.320858 21084 tablet_replica.cc:330] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2: stopping tablet replica
I20240501 08:04:59.321395 21084 raft_consensus.cc:2229] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:04:59.321931 21084 raft_consensus.cc:2258] T 6318bac57a7c4c6180f8b953946ad8c4 P 17766bfac44e4187887d8a1dddbd74d2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:59.341264 21084 tablet_server.cc:194] TabletServer@127.20.151.3:0 shutdown complete.
I20240501 08:04:59.351452 21084 master.cc:537] Master@127.20.151.62:37567 shutting down...
I20240501 08:04:59.368098 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:04:59.368687 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:04:59.369029 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8af2ee35a2574b359c35f382e7408e9e: stopping tablet replica
I20240501 08:04:59.386965 21084 master.cc:552] Master@127.20.151.62:37567 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (13221 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20240501 08:04:59.420774 21084 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.151.62:34541
I20240501 08:04:59.421833 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:59.427747 25225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:59.428768 25226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:59.429770 25228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:59.430256 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:59.431120 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:59.431331 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:59.431444 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550699431434 us; error 0 us; skew 500 ppm
I20240501 08:04:59.431918 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:59.438866 21084 webserver.cc:460] Webserver started at http://127.20.151.62:46687/ using document root <none> and password file <none>
I20240501 08:04:59.439329 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:59.439484 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:59.439715 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:59.440979 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/master-0-root/instance:
uuid: "e205c2d1549b4ab492d4ff565075239b"
format_stamp: "Formatted at 2024-05-01 08:04:59 on dist-test-slave-pwkb"
I20240501 08:04:59.445366 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:04:59.448676 25233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:59.449493 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:04:59.449822 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/master-0-root
uuid: "e205c2d1549b4ab492d4ff565075239b"
format_stamp: "Formatted at 2024-05-01 08:04:59 on dist-test-slave-pwkb"
I20240501 08:04:59.450102 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:59.468212 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:59.469372 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:59.506322 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:34541
I20240501 08:04:59.506423 25284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:34541 every 8 connection(s)
I20240501 08:04:59.510118 25285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:04:59.521252 25285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b: Bootstrap starting.
I20240501 08:04:59.525383 25285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b: Neither blocks nor log segments found. Creating new log.
I20240501 08:04:59.529498 25285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b: No bootstrap required, opened a new log
I20240501 08:04:59.531504 25285 raft_consensus.cc:348] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e205c2d1549b4ab492d4ff565075239b" member_type: VOTER }
I20240501 08:04:59.531958 25285 raft_consensus.cc:374] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:04:59.532173 25285 raft_consensus.cc:729] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e205c2d1549b4ab492d4ff565075239b, State: Initialized, Role: FOLLOWER
I20240501 08:04:59.532752 25285 consensus_queue.cc:260] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e205c2d1549b4ab492d4ff565075239b" member_type: VOTER }
I20240501 08:04:59.533215 25285 raft_consensus.cc:388] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:04:59.533437 25285 raft_consensus.cc:482] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:04:59.533715 25285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:04:59.537886 25285 raft_consensus.cc:504] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e205c2d1549b4ab492d4ff565075239b" member_type: VOTER }
I20240501 08:04:59.538344 25285 leader_election.cc:302] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e205c2d1549b4ab492d4ff565075239b; no voters:
I20240501 08:04:59.539378 25285 leader_election.cc:288] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:04:59.539763 25288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:04:59.541030 25288 raft_consensus.cc:686] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [term 1 LEADER]: Becoming Leader. State: Replica: e205c2d1549b4ab492d4ff565075239b, State: Running, Role: LEADER
I20240501 08:04:59.541607 25288 consensus_queue.cc:237] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e205c2d1549b4ab492d4ff565075239b" member_type: VOTER }
I20240501 08:04:59.542290 25285 sys_catalog.cc:564] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:04:59.546066 25289 sys_catalog.cc:455] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e205c2d1549b4ab492d4ff565075239b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e205c2d1549b4ab492d4ff565075239b" member_type: VOTER } }
I20240501 08:04:59.546903 25289 sys_catalog.cc:458] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [sys.catalog]: This master's current role is: LEADER
I20240501 08:04:59.546609 25290 sys_catalog.cc:455] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e205c2d1549b4ab492d4ff565075239b. Latest consensus state: current_term: 1 leader_uuid: "e205c2d1549b4ab492d4ff565075239b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e205c2d1549b4ab492d4ff565075239b" member_type: VOTER } }
I20240501 08:04:59.548094 25290 sys_catalog.cc:458] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [sys.catalog]: This master's current role is: LEADER
I20240501 08:04:59.554167 25296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:04:59.561674 25296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:04:59.562620 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:04:59.570318 25296 catalog_manager.cc:1347] Generated new cluster ID: 6becdc307c24456196def3c095b37084
I20240501 08:04:59.570731 25296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:04:59.601218 25296 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:04:59.602471 25296 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:04:59.617842 25296 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b: Generated new TSK 0
I20240501 08:04:59.618373 25296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:04:59.628969 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:59.635939 25306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:59.637398 25307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:59.638482 25309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:59.639828 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:59.641808 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:59.642012 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:59.642151 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550699642139 us; error 0 us; skew 500 ppm
I20240501 08:04:59.642668 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:59.645044 21084 webserver.cc:460] Webserver started at http://127.20.151.1:35385/ using document root <none> and password file <none>
I20240501 08:04:59.645447 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:59.645603 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:59.645860 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:59.646899 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-0-root/instance:
uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9"
format_stamp: "Formatted at 2024-05-01 08:04:59 on dist-test-slave-pwkb"
I20240501 08:04:59.651171 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:04:59.654439 25314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:59.655151 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:04:59.655539 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-0-root
uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9"
format_stamp: "Formatted at 2024-05-01 08:04:59 on dist-test-slave-pwkb"
I20240501 08:04:59.655912 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:59.669229 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:59.670356 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:59.671727 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:59.673959 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:59.674150 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:59.674371 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:59.674531 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:59.715106 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:45579
I20240501 08:04:59.715229 25376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:45579 every 8 connection(s)
I20240501 08:04:59.719964 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:59.727946 25381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:59.729275 25382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:59.731812 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:59.733554 25377 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34541
I20240501 08:04:59.734051 25377 heartbeater.cc:464] Registering TS with master...
W20240501 08:04:59.734086 25384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:59.735199 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 08:04:59.735199 25377 heartbeater.cc:511] Master 127.20.151.62:34541 requested a full tablet report, sending...
W20240501 08:04:59.735620 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:59.735843 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550699735827 us; error 0 us; skew 500 ppm
I20240501 08:04:59.736728 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:59.737882 25250 ts_manager.cc:194] Registered new tserver with Master: a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579)
I20240501 08:04:59.740480 21084 webserver.cc:460] Webserver started at http://127.20.151.2:43099/ using document root <none> and password file <none>
I20240501 08:04:59.740527 25250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47588
I20240501 08:04:59.741050 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:59.741279 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:59.741565 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:59.743201 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-1-root/instance:
uuid: "3725d5798b684f8d9b82b754668ae5da"
format_stamp: "Formatted at 2024-05-01 08:04:59 on dist-test-slave-pwkb"
I20240501 08:04:59.747517 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:04:59.750883 25389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:59.752017 21084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:04:59.752420 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-1-root
uuid: "3725d5798b684f8d9b82b754668ae5da"
format_stamp: "Formatted at 2024-05-01 08:04:59 on dist-test-slave-pwkb"
I20240501 08:04:59.752774 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:59.767386 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:59.768527 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:59.770104 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:59.772323 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:59.772517 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:59.772738 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:59.772889 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:59.810506 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:43465
I20240501 08:04:59.810600 25451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:43465 every 8 connection(s)
I20240501 08:04:59.815131 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:04:59.823626 25455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:59.824374 25456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:04:59.825182 25458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:04:59.827131 21084 server_base.cc:1034] running on GCE node
I20240501 08:04:59.828279 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:04:59.828477 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:04:59.828657 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550699828640 us; error 0 us; skew 500 ppm
I20240501 08:04:59.828907 25452 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34541
I20240501 08:04:59.829187 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:04:59.829229 25452 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:59.830282 25452 heartbeater.cc:511] Master 127.20.151.62:34541 requested a full tablet report, sending...
I20240501 08:04:59.832247 25250 ts_manager.cc:194] Registered new tserver with Master: 3725d5798b684f8d9b82b754668ae5da (127.20.151.2:43465)
I20240501 08:04:59.832845 21084 webserver.cc:460] Webserver started at http://127.20.151.3:34489/ using document root <none> and password file <none>
I20240501 08:04:59.833314 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:04:59.833503 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:04:59.833591 25250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47600
I20240501 08:04:59.833781 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:04:59.835129 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-2-root/instance:
uuid: "526d6b18efbb4af9a9a0216e222029b2"
format_stamp: "Formatted at 2024-05-01 08:04:59 on dist-test-slave-pwkb"
I20240501 08:04:59.839478 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240501 08:04:59.842562 25463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:59.843334 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:04:59.843600 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-2-root
uuid: "526d6b18efbb4af9a9a0216e222029b2"
format_stamp: "Formatted at 2024-05-01 08:04:59 on dist-test-slave-pwkb"
I20240501 08:04:59.843858 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:04:59.866530 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:04:59.867605 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:04:59.868945 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:04:59.871193 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:04:59.871366 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:59.871614 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:04:59.871754 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:04:59.910171 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:37755
I20240501 08:04:59.910277 25525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:37755 every 8 connection(s)
I20240501 08:04:59.923316 25526 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34541
I20240501 08:04:59.923643 25526 heartbeater.cc:464] Registering TS with master...
I20240501 08:04:59.924297 25526 heartbeater.cc:511] Master 127.20.151.62:34541 requested a full tablet report, sending...
I20240501 08:04:59.926117 25250 ts_manager.cc:194] Registered new tserver with Master: 526d6b18efbb4af9a9a0216e222029b2 (127.20.151.3:37755)
I20240501 08:04:59.926911 21084 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013634826s
I20240501 08:04:59.927469 25250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47608
I20240501 08:05:00.743372 25377 heartbeater.cc:503] Master 127.20.151.62:34541 was elected leader, sending a full tablet report...
I20240501 08:05:00.835944 25452 heartbeater.cc:503] Master 127.20.151.62:34541 was elected leader, sending a full tablet report...
I20240501 08:05:00.929803 25526 heartbeater.cc:503] Master 127.20.151.62:34541 was elected leader, sending a full tablet report...
I20240501 08:05:00.959609 21084 test_util.cc:268] Using random seed: -243603614
I20240501 08:05:00.981453 25250 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47616:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:05:00.985042 25250 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:05:01.045679 25342 tablet_service.cc:1467] Processing CreateTablet for tablet d59079b3a4f44463b337d12cf267ca54 (DEFAULT_TABLE table=test-workload [id=7a7180f894124c4b958fa1c4cf670d64]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 08:05:01.046438 25341 tablet_service.cc:1467] Processing CreateTablet for tablet 26841e9dcbca4d6d995fba9546ada63c (DEFAULT_TABLE table=test-workload [id=7a7180f894124c4b958fa1c4cf670d64]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 08:05:01.047155 25342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59079b3a4f44463b337d12cf267ca54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:01.047777 25341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26841e9dcbca4d6d995fba9546ada63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:01.051059 25340 tablet_service.cc:1467] Processing CreateTablet for tablet f652910a32a348339c345ce75ce79be7 (DEFAULT_TABLE table=test-workload [id=7a7180f894124c4b958fa1c4cf670d64]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 08:05:01.051844 25339 tablet_service.cc:1467] Processing CreateTablet for tablet 19ccc66f74df4b3ebc1fceb2499c0308 (DEFAULT_TABLE table=test-workload [id=7a7180f894124c4b958fa1c4cf670d64]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 08:05:01.052419 25340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f652910a32a348339c345ce75ce79be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:01.052704 25417 tablet_service.cc:1467] Processing CreateTablet for tablet d59079b3a4f44463b337d12cf267ca54 (DEFAULT_TABLE table=test-workload [id=7a7180f894124c4b958fa1c4cf670d64]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 08:05:01.054054 25417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59079b3a4f44463b337d12cf267ca54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:01.058883 25416 tablet_service.cc:1467] Processing CreateTablet for tablet 26841e9dcbca4d6d995fba9546ada63c (DEFAULT_TABLE table=test-workload [id=7a7180f894124c4b958fa1c4cf670d64]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 08:05:01.060227 25416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26841e9dcbca4d6d995fba9546ada63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:01.053125 25339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ccc66f74df4b3ebc1fceb2499c0308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:01.069990 25415 tablet_service.cc:1467] Processing CreateTablet for tablet f652910a32a348339c345ce75ce79be7 (DEFAULT_TABLE table=test-workload [id=7a7180f894124c4b958fa1c4cf670d64]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 08:05:01.071853 25415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f652910a32a348339c345ce75ce79be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:01.074687 25414 tablet_service.cc:1467] Processing CreateTablet for tablet 19ccc66f74df4b3ebc1fceb2499c0308 (DEFAULT_TABLE table=test-workload [id=7a7180f894124c4b958fa1c4cf670d64]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 08:05:01.076388 25414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ccc66f74df4b3ebc1fceb2499c0308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:01.096237 25491 tablet_service.cc:1467] Processing CreateTablet for tablet d59079b3a4f44463b337d12cf267ca54 (DEFAULT_TABLE table=test-workload [id=7a7180f894124c4b958fa1c4cf670d64]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 08:05:01.099269 25490 tablet_service.cc:1467] Processing CreateTablet for tablet 26841e9dcbca4d6d995fba9546ada63c (DEFAULT_TABLE table=test-workload [id=7a7180f894124c4b958fa1c4cf670d64]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 08:05:01.103184 25488 tablet_service.cc:1467] Processing CreateTablet for tablet 19ccc66f74df4b3ebc1fceb2499c0308 (DEFAULT_TABLE table=test-workload [id=7a7180f894124c4b958fa1c4cf670d64]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 08:05:01.105212 25489 tablet_service.cc:1467] Processing CreateTablet for tablet f652910a32a348339c345ce75ce79be7 (DEFAULT_TABLE table=test-workload [id=7a7180f894124c4b958fa1c4cf670d64]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 08:05:01.107023 25491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59079b3a4f44463b337d12cf267ca54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:01.107478 25546 tablet_bootstrap.cc:492] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9: Bootstrap starting.
I20240501 08:05:01.107906 25490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26841e9dcbca4d6d995fba9546ada63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:01.108561 25488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ccc66f74df4b3ebc1fceb2499c0308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:01.109215 25489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f652910a32a348339c345ce75ce79be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:01.115243 25546 tablet_bootstrap.cc:654] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:01.190078 25547 tablet_bootstrap.cc:492] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da: Bootstrap starting.
I20240501 08:05:01.195720 25547 tablet_bootstrap.cc:654] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:01.196962 25546 tablet_bootstrap.cc:492] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9: No bootstrap required, opened a new log
I20240501 08:05:01.197384 25549 tablet_bootstrap.cc:492] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2: Bootstrap starting.
I20240501 08:05:01.198484 25546 ts_tablet_manager.cc:1367] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9: Time spent bootstrapping tablet: real 0.091s user 0.040s sys 0.027s
I20240501 08:05:01.201419 25546 raft_consensus.cc:348] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.202236 25546 raft_consensus.cc:374] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:01.202373 25547 tablet_bootstrap.cc:492] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da: No bootstrap required, opened a new log
I20240501 08:05:01.202562 25546 raft_consensus.cc:729] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a4ec80f9c949ddb15d9d10afb32cc9, State: Initialized, Role: FOLLOWER
I20240501 08:05:01.202812 25547 ts_tablet_manager.cc:1367] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240501 08:05:01.203531 25549 tablet_bootstrap.cc:654] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:01.203356 25546 consensus_queue.cc:260] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.204993 25547 raft_consensus.cc:348] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.205966 25547 raft_consensus.cc:374] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:01.206220 25547 raft_consensus.cc:729] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3725d5798b684f8d9b82b754668ae5da, State: Initialized, Role: FOLLOWER
I20240501 08:05:01.206784 25547 consensus_queue.cc:260] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.208163 25546 ts_tablet_manager.cc:1398] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240501 08:05:01.209182 25546 tablet_bootstrap.cc:492] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9: Bootstrap starting.
I20240501 08:05:01.210686 25547 ts_tablet_manager.cc:1398] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240501 08:05:01.211663 25547 tablet_bootstrap.cc:492] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da: Bootstrap starting.
I20240501 08:05:01.215078 25546 tablet_bootstrap.cc:654] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:01.216936 25547 tablet_bootstrap.cc:654] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:01.223302 25549 tablet_bootstrap.cc:492] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2: No bootstrap required, opened a new log
I20240501 08:05:01.223909 25549 ts_tablet_manager.cc:1367] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.000s
I20240501 08:05:01.227449 25549 raft_consensus.cc:348] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.228439 25549 raft_consensus.cc:374] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:01.228802 25549 raft_consensus.cc:729] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526d6b18efbb4af9a9a0216e222029b2, State: Initialized, Role: FOLLOWER
I20240501 08:05:01.229727 25549 consensus_queue.cc:260] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.232401 25547 tablet_bootstrap.cc:492] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da: No bootstrap required, opened a new log
I20240501 08:05:01.232789 25547 ts_tablet_manager.cc:1367] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.012s
I20240501 08:05:01.234033 25546 tablet_bootstrap.cc:492] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9: No bootstrap required, opened a new log
I20240501 08:05:01.234535 25546 ts_tablet_manager.cc:1367] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.015s
I20240501 08:05:01.235051 25549 ts_tablet_manager.cc:1398] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20240501 08:05:01.235105 25547 raft_consensus.cc:348] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.235976 25547 raft_consensus.cc:374] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:01.236099 25549 tablet_bootstrap.cc:492] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2: Bootstrap starting.
I20240501 08:05:01.236279 25547 raft_consensus.cc:729] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3725d5798b684f8d9b82b754668ae5da, State: Initialized, Role: FOLLOWER
I20240501 08:05:01.237010 25547 consensus_queue.cc:260] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.237059 25546 raft_consensus.cc:348] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.238049 25546 raft_consensus.cc:374] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:01.238382 25546 raft_consensus.cc:729] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a4ec80f9c949ddb15d9d10afb32cc9, State: Initialized, Role: FOLLOWER
I20240501 08:05:01.239529 25547 ts_tablet_manager.cc:1398] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240501 08:05:01.239193 25546 consensus_queue.cc:260] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.240469 25547 tablet_bootstrap.cc:492] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da: Bootstrap starting.
I20240501 08:05:01.241812 25546 ts_tablet_manager.cc:1398] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:05:01.242904 25546 tablet_bootstrap.cc:492] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9: Bootstrap starting.
I20240501 08:05:01.246644 25547 tablet_bootstrap.cc:654] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:01.248836 25546 tablet_bootstrap.cc:654] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:01.251531 25547 tablet_bootstrap.cc:492] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da: No bootstrap required, opened a new log
I20240501 08:05:01.252019 25547 ts_tablet_manager.cc:1367] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.001s
I20240501 08:05:01.254832 25546 tablet_bootstrap.cc:492] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9: No bootstrap required, opened a new log
I20240501 08:05:01.254772 25547 raft_consensus.cc:348] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.255474 25546 ts_tablet_manager.cc:1367] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20240501 08:05:01.255731 25547 raft_consensus.cc:374] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:01.256043 25547 raft_consensus.cc:729] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3725d5798b684f8d9b82b754668ae5da, State: Initialized, Role: FOLLOWER
I20240501 08:05:01.256778 25547 consensus_queue.cc:260] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.257853 25546 raft_consensus.cc:348] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.258390 25546 raft_consensus.cc:374] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:01.258601 25546 raft_consensus.cc:729] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a4ec80f9c949ddb15d9d10afb32cc9, State: Initialized, Role: FOLLOWER
I20240501 08:05:01.259254 25547 ts_tablet_manager.cc:1398] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:05:01.259279 25546 consensus_queue.cc:260] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.260349 25547 tablet_bootstrap.cc:492] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da: Bootstrap starting.
I20240501 08:05:01.260702 25549 tablet_bootstrap.cc:654] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:01.261595 25546 ts_tablet_manager.cc:1398] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20240501 08:05:01.262583 25546 tablet_bootstrap.cc:492] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9: Bootstrap starting.
I20240501 08:05:01.266836 25547 tablet_bootstrap.cc:654] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:01.268014 25546 tablet_bootstrap.cc:654] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:01.275875 25549 tablet_bootstrap.cc:492] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2: No bootstrap required, opened a new log
I20240501 08:05:01.276512 25549 ts_tablet_manager.cc:1367] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.004s
I20240501 08:05:01.279856 25549 raft_consensus.cc:348] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.280843 25549 raft_consensus.cc:374] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:01.281206 25549 raft_consensus.cc:729] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526d6b18efbb4af9a9a0216e222029b2, State: Initialized, Role: FOLLOWER
I20240501 08:05:01.282310 25549 consensus_queue.cc:260] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.287467 25549 ts_tablet_manager.cc:1398] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20240501 08:05:01.287907 25547 tablet_bootstrap.cc:492] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da: No bootstrap required, opened a new log
I20240501 08:05:01.288367 25549 tablet_bootstrap.cc:492] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2: Bootstrap starting.
I20240501 08:05:01.288393 25547 ts_tablet_manager.cc:1367] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.012s
I20240501 08:05:01.291225 25547 raft_consensus.cc:348] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.291992 25547 raft_consensus.cc:374] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:01.292285 25547 raft_consensus.cc:729] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3725d5798b684f8d9b82b754668ae5da, State: Initialized, Role: FOLLOWER
I20240501 08:05:01.293015 25547 consensus_queue.cc:260] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.295380 25547 ts_tablet_manager.cc:1398] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240501 08:05:01.299278 25549 tablet_bootstrap.cc:654] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:01.300772 25546 tablet_bootstrap.cc:492] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9: No bootstrap required, opened a new log
I20240501 08:05:01.301170 25546 ts_tablet_manager.cc:1367] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.020s
I20240501 08:05:01.303786 25546 raft_consensus.cc:348] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.304368 25546 raft_consensus.cc:374] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:01.304450 25549 tablet_bootstrap.cc:492] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2: No bootstrap required, opened a new log
I20240501 08:05:01.304623 25546 raft_consensus.cc:729] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a4ec80f9c949ddb15d9d10afb32cc9, State: Initialized, Role: FOLLOWER
I20240501 08:05:01.305004 25549 ts_tablet_manager.cc:1367] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.008s
I20240501 08:05:01.305320 25546 consensus_queue.cc:260] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.309338 25546 ts_tablet_manager.cc:1398] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240501 08:05:01.309245 25549 raft_consensus.cc:348] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.309890 25549 raft_consensus.cc:374] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:01.310212 25549 raft_consensus.cc:729] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526d6b18efbb4af9a9a0216e222029b2, State: Initialized, Role: FOLLOWER
I20240501 08:05:01.310861 25549 consensus_queue.cc:260] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.312480 25549 ts_tablet_manager.cc:1398] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240501 08:05:01.313334 25549 tablet_bootstrap.cc:492] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2: Bootstrap starting.
I20240501 08:05:01.318235 25549 tablet_bootstrap.cc:654] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:01.323060 25549 tablet_bootstrap.cc:492] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2: No bootstrap required, opened a new log
I20240501 08:05:01.323400 25549 ts_tablet_manager.cc:1367] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240501 08:05:01.325239 25549 raft_consensus.cc:348] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.325750 25549 raft_consensus.cc:374] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:01.325958 25549 raft_consensus.cc:729] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526d6b18efbb4af9a9a0216e222029b2, State: Initialized, Role: FOLLOWER
I20240501 08:05:01.326470 25549 consensus_queue.cc:260] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.328119 25549 ts_tablet_manager.cc:1398] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20240501 08:05:01.334955 25556 raft_consensus.cc:482] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:01.335431 25556 raft_consensus.cc:504] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.337431 25556 leader_election.cc:288] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579), 3725d5798b684f8d9b82b754668ae5da (127.20.151.2:43465)
I20240501 08:05:01.347752 25352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ccc66f74df4b3ebc1fceb2499c0308" candidate_uuid: "526d6b18efbb4af9a9a0216e222029b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" is_pre_election: true
I20240501 08:05:01.348215 25427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ccc66f74df4b3ebc1fceb2499c0308" candidate_uuid: "526d6b18efbb4af9a9a0216e222029b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3725d5798b684f8d9b82b754668ae5da" is_pre_election: true
I20240501 08:05:01.348569 25352 raft_consensus.cc:2454] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 526d6b18efbb4af9a9a0216e222029b2 in term 0.
I20240501 08:05:01.348824 25427 raft_consensus.cc:2454] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 526d6b18efbb4af9a9a0216e222029b2 in term 0.
I20240501 08:05:01.349709 25466 leader_election.cc:302] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526d6b18efbb4af9a9a0216e222029b2, a6a4ec80f9c949ddb15d9d10afb32cc9; no voters:
I20240501 08:05:01.350363 25556 raft_consensus.cc:2781] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:05:01.350701 25556 raft_consensus.cc:482] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:01.351011 25556 raft_consensus.cc:3037] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:01.355829 25556 raft_consensus.cc:504] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.357275 25556 leader_election.cc:288] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 1 election: Requested vote from peers a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579), 3725d5798b684f8d9b82b754668ae5da (127.20.151.2:43465)
I20240501 08:05:01.358042 25352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ccc66f74df4b3ebc1fceb2499c0308" candidate_uuid: "526d6b18efbb4af9a9a0216e222029b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9"
I20240501 08:05:01.358234 25427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ccc66f74df4b3ebc1fceb2499c0308" candidate_uuid: "526d6b18efbb4af9a9a0216e222029b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3725d5798b684f8d9b82b754668ae5da"
I20240501 08:05:01.358642 25352 raft_consensus.cc:3037] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:01.358738 25427 raft_consensus.cc:3037] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:01.365473 25427 raft_consensus.cc:2454] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 526d6b18efbb4af9a9a0216e222029b2 in term 1.
I20240501 08:05:01.365473 25352 raft_consensus.cc:2454] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 526d6b18efbb4af9a9a0216e222029b2 in term 1.
I20240501 08:05:01.366662 25466 leader_election.cc:302] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526d6b18efbb4af9a9a0216e222029b2, a6a4ec80f9c949ddb15d9d10afb32cc9; no voters:
I20240501 08:05:01.367234 25556 raft_consensus.cc:2781] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:05:01.368204 25556 raft_consensus.cc:686] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 LEADER]: Becoming Leader. State: Replica: 526d6b18efbb4af9a9a0216e222029b2, State: Running, Role: LEADER
I20240501 08:05:01.368799 25556 consensus_queue.cc:237] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.375871 25247 catalog_manager.cc:5484] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 526d6b18efbb4af9a9a0216e222029b2 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "526d6b18efbb4af9a9a0216e222029b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: HEALTHY } } }
I20240501 08:05:01.392758 25556 raft_consensus.cc:482] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:01.393191 25556 raft_consensus.cc:504] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.394832 25556 leader_election.cc:288] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579), 3725d5798b684f8d9b82b754668ae5da (127.20.151.2:43465)
I20240501 08:05:01.395594 25352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26841e9dcbca4d6d995fba9546ada63c" candidate_uuid: "526d6b18efbb4af9a9a0216e222029b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" is_pre_election: true
I20240501 08:05:01.395849 25427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26841e9dcbca4d6d995fba9546ada63c" candidate_uuid: "526d6b18efbb4af9a9a0216e222029b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3725d5798b684f8d9b82b754668ae5da" is_pre_election: true
I20240501 08:05:01.396283 25352 raft_consensus.cc:2454] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 526d6b18efbb4af9a9a0216e222029b2 in term 0.
I20240501 08:05:01.396466 25427 raft_consensus.cc:2454] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 526d6b18efbb4af9a9a0216e222029b2 in term 0.
I20240501 08:05:01.397243 25466 leader_election.cc:302] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526d6b18efbb4af9a9a0216e222029b2, a6a4ec80f9c949ddb15d9d10afb32cc9; no voters:
I20240501 08:05:01.397837 25556 raft_consensus.cc:2781] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:05:01.398114 25556 raft_consensus.cc:482] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:01.398427 25556 raft_consensus.cc:3037] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:01.402861 25556 raft_consensus.cc:504] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.404368 25556 leader_election.cc:288] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 1 election: Requested vote from peers a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579), 3725d5798b684f8d9b82b754668ae5da (127.20.151.2:43465)
I20240501 08:05:01.405164 25352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26841e9dcbca4d6d995fba9546ada63c" candidate_uuid: "526d6b18efbb4af9a9a0216e222029b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9"
I20240501 08:05:01.405475 25427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26841e9dcbca4d6d995fba9546ada63c" candidate_uuid: "526d6b18efbb4af9a9a0216e222029b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3725d5798b684f8d9b82b754668ae5da"
I20240501 08:05:01.405802 25352 raft_consensus.cc:3037] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:01.406054 25427 raft_consensus.cc:3037] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:01.412178 25352 raft_consensus.cc:2454] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 526d6b18efbb4af9a9a0216e222029b2 in term 1.
I20240501 08:05:01.412173 25427 raft_consensus.cc:2454] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 526d6b18efbb4af9a9a0216e222029b2 in term 1.
I20240501 08:05:01.413352 25466 leader_election.cc:302] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526d6b18efbb4af9a9a0216e222029b2, a6a4ec80f9c949ddb15d9d10afb32cc9; no voters:
I20240501 08:05:01.414063 25556 raft_consensus.cc:2781] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:05:01.414479 25556 raft_consensus.cc:686] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 1 LEADER]: Becoming Leader. State: Replica: 526d6b18efbb4af9a9a0216e222029b2, State: Running, Role: LEADER
I20240501 08:05:01.415226 25556 consensus_queue.cc:237] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.421314 25247 catalog_manager.cc:5484] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 526d6b18efbb4af9a9a0216e222029b2 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "526d6b18efbb4af9a9a0216e222029b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: HEALTHY } } }
I20240501 08:05:01.486091 25553 raft_consensus.cc:482] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:01.486518 25553 raft_consensus.cc:504] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.488253 25553 leader_election.cc:288] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579), 526d6b18efbb4af9a9a0216e222029b2 (127.20.151.3:37755)
I20240501 08:05:01.498039 25352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f652910a32a348339c345ce75ce79be7" candidate_uuid: "3725d5798b684f8d9b82b754668ae5da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" is_pre_election: true
I20240501 08:05:01.498550 25501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f652910a32a348339c345ce75ce79be7" candidate_uuid: "3725d5798b684f8d9b82b754668ae5da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526d6b18efbb4af9a9a0216e222029b2" is_pre_election: true
I20240501 08:05:01.498755 25352 raft_consensus.cc:2454] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3725d5798b684f8d9b82b754668ae5da in term 0.
I20240501 08:05:01.499239 25501 raft_consensus.cc:2454] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3725d5798b684f8d9b82b754668ae5da in term 0.
I20240501 08:05:01.499791 25392 leader_election.cc:302] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3725d5798b684f8d9b82b754668ae5da, a6a4ec80f9c949ddb15d9d10afb32cc9; no voters:
I20240501 08:05:01.500543 25553 raft_consensus.cc:2781] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:05:01.500794 25553 raft_consensus.cc:482] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:01.501053 25553 raft_consensus.cc:3037] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:01.505489 25553 raft_consensus.cc:504] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.506835 25553 leader_election.cc:288] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [CANDIDATE]: Term 1 election: Requested vote from peers a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579), 526d6b18efbb4af9a9a0216e222029b2 (127.20.151.3:37755)
I20240501 08:05:01.507539 25352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f652910a32a348339c345ce75ce79be7" candidate_uuid: "3725d5798b684f8d9b82b754668ae5da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9"
I20240501 08:05:01.507781 25501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f652910a32a348339c345ce75ce79be7" candidate_uuid: "3725d5798b684f8d9b82b754668ae5da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526d6b18efbb4af9a9a0216e222029b2"
I20240501 08:05:01.508126 25352 raft_consensus.cc:3037] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:01.508313 25501 raft_consensus.cc:3037] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:01.514652 25501 raft_consensus.cc:2454] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3725d5798b684f8d9b82b754668ae5da in term 1.
I20240501 08:05:01.514652 25352 raft_consensus.cc:2454] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3725d5798b684f8d9b82b754668ae5da in term 1.
I20240501 08:05:01.515885 25392 leader_election.cc:302] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3725d5798b684f8d9b82b754668ae5da, a6a4ec80f9c949ddb15d9d10afb32cc9; no voters:
I20240501 08:05:01.516527 25553 raft_consensus.cc:2781] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:05:01.517489 25553 raft_consensus.cc:686] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 1 LEADER]: Becoming Leader. State: Replica: 3725d5798b684f8d9b82b754668ae5da, State: Running, Role: LEADER
I20240501 08:05:01.518126 25553 consensus_queue.cc:237] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.524222 25247 catalog_manager.cc:5484] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da reported cstate change: term changed from 0 to 1, leader changed from <none> to 3725d5798b684f8d9b82b754668ae5da (127.20.151.2). New cstate: current_term: 1 leader_uuid: "3725d5798b684f8d9b82b754668ae5da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:01.552364 25551 raft_consensus.cc:482] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:01.552835 25551 raft_consensus.cc:504] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
W20240501 08:05:01.554775 25298 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 08:05:01.555083 25551 leader_election.cc:288] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3725d5798b684f8d9b82b754668ae5da (127.20.151.2:43465), 526d6b18efbb4af9a9a0216e222029b2 (127.20.151.3:37755)
I20240501 08:05:01.564347 25427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59079b3a4f44463b337d12cf267ca54" candidate_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3725d5798b684f8d9b82b754668ae5da" is_pre_election: true
I20240501 08:05:01.565007 25427 raft_consensus.cc:2454] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a4ec80f9c949ddb15d9d10afb32cc9 in term 0.
I20240501 08:05:01.565557 25501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59079b3a4f44463b337d12cf267ca54" candidate_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526d6b18efbb4af9a9a0216e222029b2" is_pre_election: true
I20240501 08:05:01.565964 25317 leader_election.cc:302] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3725d5798b684f8d9b82b754668ae5da, a6a4ec80f9c949ddb15d9d10afb32cc9; no voters:
I20240501 08:05:01.566067 25501 raft_consensus.cc:2454] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a4ec80f9c949ddb15d9d10afb32cc9 in term 0.
I20240501 08:05:01.566623 25551 raft_consensus.cc:2781] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:05:01.566983 25551 raft_consensus.cc:482] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:01.567328 25551 raft_consensus.cc:3037] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:01.571837 25551 raft_consensus.cc:504] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.573295 25551 leader_election.cc:288] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [CANDIDATE]: Term 1 election: Requested vote from peers 3725d5798b684f8d9b82b754668ae5da (127.20.151.2:43465), 526d6b18efbb4af9a9a0216e222029b2 (127.20.151.3:37755)
I20240501 08:05:01.574012 25427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59079b3a4f44463b337d12cf267ca54" candidate_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3725d5798b684f8d9b82b754668ae5da"
I20240501 08:05:01.574265 25501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59079b3a4f44463b337d12cf267ca54" candidate_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526d6b18efbb4af9a9a0216e222029b2"
I20240501 08:05:01.574620 25427 raft_consensus.cc:3037] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:01.574788 25501 raft_consensus.cc:3037] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:01.581075 25501 raft_consensus.cc:2454] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a4ec80f9c949ddb15d9d10afb32cc9 in term 1.
I20240501 08:05:01.581382 25427 raft_consensus.cc:2454] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a4ec80f9c949ddb15d9d10afb32cc9 in term 1.
I20240501 08:05:01.582274 25316 leader_election.cc:302] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526d6b18efbb4af9a9a0216e222029b2, a6a4ec80f9c949ddb15d9d10afb32cc9; no voters:
I20240501 08:05:01.582907 25551 raft_consensus.cc:2781] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:05:01.583715 25551 raft_consensus.cc:686] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 LEADER]: Becoming Leader. State: Replica: a6a4ec80f9c949ddb15d9d10afb32cc9, State: Running, Role: LEADER
I20240501 08:05:01.584398 25551 consensus_queue.cc:237] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:01.590679 25247 catalog_manager.cc:5484] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:01.605382 21084 tablet_server.cc:178] TabletServer@127.20.151.1:0 shutting down...
I20240501 08:05:01.620797 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:05:01.621497 21084 tablet_replica.cc:330] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9: stopping tablet replica
I20240501 08:05:01.622032 21084 raft_consensus.cc:2229] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:01.622404 21084 raft_consensus.cc:2258] T f652910a32a348339c345ce75ce79be7 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:01.624235 21084 tablet_replica.cc:330] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9: stopping tablet replica
I20240501 08:05:01.624708 21084 raft_consensus.cc:2229] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:01.625041 21084 raft_consensus.cc:2258] T 19ccc66f74df4b3ebc1fceb2499c0308 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:01.626756 21084 tablet_replica.cc:330] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9: stopping tablet replica
I20240501 08:05:01.627213 21084 raft_consensus.cc:2229] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:01.627584 21084 raft_consensus.cc:2258] T 26841e9dcbca4d6d995fba9546ada63c P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:01.629262 21084 tablet_replica.cc:330] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9: stopping tablet replica
I20240501 08:05:01.629736 21084 raft_consensus.cc:2229] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:05:01.630249 21084 pending_rounds.cc:62] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9: Trying to abort 1 pending ops.
I20240501 08:05:01.630452 21084 pending_rounds.cc:68] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022799673696411648 op_type: NO_OP noop_request { }
I20240501 08:05:01.630745 21084 raft_consensus.cc:2866] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 08:05:01.631008 21084 raft_consensus.cc:2258] T d59079b3a4f44463b337d12cf267ca54 P a6a4ec80f9c949ddb15d9d10afb32cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:01.652709 21084 tablet_server.cc:194] TabletServer@127.20.151.1:0 shutdown complete.
I20240501 08:05:01.829989 25556 consensus_queue.cc:1035] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:05:01.840080 25466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240501 08:05:01.843734 25466 consensus_peers.cc:485] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 -> Peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Couldn't send request to peer a6a4ec80f9c949ddb15d9d10afb32cc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:01.926575 25556 consensus_queue.cc:1035] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:05:01.939874 25466 consensus_peers.cc:485] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 -> Peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Couldn't send request to peer a6a4ec80f9c949ddb15d9d10afb32cc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:02.033978 25553 consensus_queue.cc:1035] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [LEADER]: Connected to new peer: Peer: permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:05:02.051393 25392 consensus_peers.cc:485] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da -> Peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Couldn't send request to peer a6a4ec80f9c949ddb15d9d10afb32cc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:02.592528 25587 consensus_queue.cc:579] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [LEADER]: Leader has been unable to successfully communicate with peer a6a4ec80f9c949ddb15d9d10afb32cc9 for more than 1 seconds (1.073s)
W20240501 08:05:02.599665 25249 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f652910a32a348339c345ce75ce79be7 with cas_config_opid_index -1: no extra replica candidate found for tablet f652910a32a348339c345ce75ce79be7 (table test-workload [id=7a7180f894124c4b958fa1c4cf670d64]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20240501 08:05:02.664103 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:05:02.670413 25592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:02.671523 25593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:02.672168 25595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:05:02.673003 21084 server_base.cc:1034] running on GCE node
I20240501 08:05:02.674024 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:05:02.674229 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:05:02.674387 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550702674370 us; error 0 us; skew 500 ppm
I20240501 08:05:02.674885 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:05:02.677126 21084 webserver.cc:460] Webserver started at http://127.20.151.4:44021/ using document root <none> and password file <none>
I20240501 08:05:02.677554 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:05:02.677762 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:05:02.678031 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:05:02.679034 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-3-root/instance:
uuid: "32544f993fc24d128352e8f4f04d5eeb"
format_stamp: "Formatted at 2024-05-01 08:05:02 on dist-test-slave-pwkb"
I20240501 08:05:02.683192 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:05:02.686101 25600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:02.686796 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:05:02.687065 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-3-root
uuid: "32544f993fc24d128352e8f4f04d5eeb"
format_stamp: "Formatted at 2024-05-01 08:05:02 on dist-test-slave-pwkb"
I20240501 08:05:02.687323 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714550573434241-21084-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:05:02.705436 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:05:02.706379 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:05:02.707685 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:05:02.709821 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:05:02.710019 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:02.710229 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:05:02.710377 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:02.746443 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.4:46233
I20240501 08:05:02.746526 25662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.4:46233 every 8 connection(s)
I20240501 08:05:02.759716 25663 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34541
I20240501 08:05:02.760071 25663 heartbeater.cc:464] Registering TS with master...
I20240501 08:05:02.760739 25663 heartbeater.cc:511] Master 127.20.151.62:34541 requested a full tablet report, sending...
I20240501 08:05:02.762634 25249 ts_manager.cc:194] Registered new tserver with Master: 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4:46233)
I20240501 08:05:02.763895 25249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47630
I20240501 08:05:02.783190 25582 consensus_queue.cc:579] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Leader has been unable to successfully communicate with peer a6a4ec80f9c949ddb15d9d10afb32cc9 for more than 1 seconds (1.367s)
I20240501 08:05:02.792574 25501 consensus_queue.cc:237] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } }
I20240501 08:05:02.798055 25427 raft_consensus.cc:1261] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:05:02.799851 25667 consensus_queue.cc:1035] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:05:02.802361 25466 consensus_peers.cc:485] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 -> Peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Couldn't send request to peer a6a4ec80f9c949ddb15d9d10afb32cc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:02.806034 25668 raft_consensus.cc:2932] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } } }
W20240501 08:05:02.808115 25464 consensus_peers.cc:485] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 -> Peer 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4:46233): Couldn't send request to peer 32544f993fc24d128352e8f4f04d5eeb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26841e9dcbca4d6d995fba9546ada63c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:02.808182 25427 raft_consensus.cc:2932] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } } }
I20240501 08:05:02.815644 25235 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 26841e9dcbca4d6d995fba9546ada63c with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:02.820075 25249 catalog_manager.cc:5484] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 reported cstate change: config changed from index -1 to 2, NON_VOTER 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) added. New cstate: current_term: 1 leader_uuid: "526d6b18efbb4af9a9a0216e222029b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:02.820744 25427 consensus_queue.cc:237] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } }
I20240501 08:05:02.828125 25501 raft_consensus.cc:1261] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Refusing update from remote peer 3725d5798b684f8d9b82b754668ae5da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:05:02.830029 25589 consensus_queue.cc:1035] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [LEADER]: Connected to new peer: Peer: permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 08:05:02.839020 25589 raft_consensus.cc:2932] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } } }
I20240501 08:05:02.840510 25501 raft_consensus.cc:2932] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } } }
W20240501 08:05:02.841814 25390 consensus_peers.cc:485] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da -> Peer 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4:46233): Couldn't send request to peer 32544f993fc24d128352e8f4f04d5eeb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f652910a32a348339c345ce75ce79be7. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:02.843641 25392 consensus_peers.cc:485] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da -> Peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Couldn't send request to peer a6a4ec80f9c949ddb15d9d10afb32cc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:02.847779 25236 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f652910a32a348339c345ce75ce79be7 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20240501 08:05:02.857887 25582 consensus_queue.cc:579] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Leader has been unable to successfully communicate with peer a6a4ec80f9c949ddb15d9d10afb32cc9 for more than 1 seconds (1.488s)
I20240501 08:05:02.857625 25249 catalog_manager.cc:5484] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da reported cstate change: config changed from index -1 to 2, NON_VOTER 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) added. New cstate: current_term: 1 leader_uuid: "3725d5798b684f8d9b82b754668ae5da" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:02.875067 25501 consensus_queue.cc:237] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } }
I20240501 08:05:02.880740 25427 raft_consensus.cc:1261] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:05:02.881181 25464 consensus_peers.cc:485] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 -> Peer 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4:46233): Couldn't send request to peer 32544f993fc24d128352e8f4f04d5eeb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19ccc66f74df4b3ebc1fceb2499c0308. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:02.882186 25668 consensus_queue.cc:1035] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:05:02.883358 25466 consensus_peers.cc:485] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 -> Peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Couldn't send request to peer a6a4ec80f9c949ddb15d9d10afb32cc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:02.888327 25667 raft_consensus.cc:2932] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } } }
I20240501 08:05:02.889815 25427 raft_consensus.cc:2932] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } } }
I20240501 08:05:02.895498 25235 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 19ccc66f74df4b3ebc1fceb2499c0308 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:02.898876 25248 catalog_manager.cc:5484] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 reported cstate change: config changed from index -1 to 2, NON_VOTER 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) added. New cstate: current_term: 1 leader_uuid: "526d6b18efbb4af9a9a0216e222029b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:03.255723 25667 raft_consensus.cc:482] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:03.256104 25667 raft_consensus.cc:504] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:03.257424 25667 leader_election.cc:288] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579), 3725d5798b684f8d9b82b754668ae5da (127.20.151.2:43465)
I20240501 08:05:03.259080 25427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59079b3a4f44463b337d12cf267ca54" candidate_uuid: "526d6b18efbb4af9a9a0216e222029b2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3725d5798b684f8d9b82b754668ae5da" is_pre_election: true
I20240501 08:05:03.259656 25427 raft_consensus.cc:2454] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 526d6b18efbb4af9a9a0216e222029b2 in term 1.
I20240501 08:05:03.260514 25466 leader_election.cc:302] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3725d5798b684f8d9b82b754668ae5da, 526d6b18efbb4af9a9a0216e222029b2; no voters:
W20240501 08:05:03.261159 25466 leader_election.cc:334] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111)
I20240501 08:05:03.261183 25667 raft_consensus.cc:2781] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:05:03.261626 25667 raft_consensus.cc:482] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:03.261937 25667 raft_consensus.cc:3037] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:03.266638 25667 raft_consensus.cc:504] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:03.267975 25667 leader_election.cc:288] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 2 election: Requested vote from peers a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579), 3725d5798b684f8d9b82b754668ae5da (127.20.151.2:43465)
I20240501 08:05:03.269348 25427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59079b3a4f44463b337d12cf267ca54" candidate_uuid: "526d6b18efbb4af9a9a0216e222029b2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3725d5798b684f8d9b82b754668ae5da"
I20240501 08:05:03.269968 25427 raft_consensus.cc:3037] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:05:03.271016 25466 leader_election.cc:334] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111)
I20240501 08:05:03.274737 25427 raft_consensus.cc:2454] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 526d6b18efbb4af9a9a0216e222029b2 in term 2.
I20240501 08:05:03.275604 25466 leader_election.cc:302] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3725d5798b684f8d9b82b754668ae5da, 526d6b18efbb4af9a9a0216e222029b2; no voters: a6a4ec80f9c949ddb15d9d10afb32cc9
I20240501 08:05:03.276476 25667 raft_consensus.cc:2781] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:05:03.276908 25678 ts_tablet_manager.cc:902] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb: Initiating tablet copy from peer 3725d5798b684f8d9b82b754668ae5da (127.20.151.2:43465)
I20240501 08:05:03.276980 25667 raft_consensus.cc:686] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 2 LEADER]: Becoming Leader. State: Replica: 526d6b18efbb4af9a9a0216e222029b2, State: Running, Role: LEADER
I20240501 08:05:03.277915 25667 consensus_queue.cc:237] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } }
I20240501 08:05:03.280524 25678 tablet_copy_client.cc:308] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.2:43465
I20240501 08:05:03.288256 25249 catalog_manager.cc:5484] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 reported cstate change: term changed from 1 to 2, leader changed from a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) to 526d6b18efbb4af9a9a0216e222029b2 (127.20.151.3). New cstate: current_term: 2 leader_uuid: "526d6b18efbb4af9a9a0216e222029b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: HEALTHY } } }
I20240501 08:05:03.296046 25437 tablet_copy_service.cc:140] P 3725d5798b684f8d9b82b754668ae5da: Received BeginTabletCopySession request for tablet f652910a32a348339c345ce75ce79be7 from peer 32544f993fc24d128352e8f4f04d5eeb ({username='slave'} at 127.0.0.1:40376)
I20240501 08:05:03.296730 25437 tablet_copy_service.cc:161] P 3725d5798b684f8d9b82b754668ae5da: Beginning new tablet copy session on tablet f652910a32a348339c345ce75ce79be7 from peer 32544f993fc24d128352e8f4f04d5eeb at {username='slave'} at 127.0.0.1:40376: session id = 32544f993fc24d128352e8f4f04d5eeb-f652910a32a348339c345ce75ce79be7
I20240501 08:05:03.302281 25686 ts_tablet_manager.cc:902] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb: Initiating tablet copy from peer 526d6b18efbb4af9a9a0216e222029b2 (127.20.151.3:37755)
I20240501 08:05:03.305147 25437 tablet_copy_source_session.cc:205] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:05:03.305763 25686 tablet_copy_client.cc:308] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.3:37755
I20240501 08:05:03.308316 25678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f652910a32a348339c345ce75ce79be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:03.316052 25511 tablet_copy_service.cc:140] P 526d6b18efbb4af9a9a0216e222029b2: Received BeginTabletCopySession request for tablet 26841e9dcbca4d6d995fba9546ada63c from peer 32544f993fc24d128352e8f4f04d5eeb ({username='slave'} at 127.0.0.1:46926)
I20240501 08:05:03.316495 25511 tablet_copy_service.cc:161] P 526d6b18efbb4af9a9a0216e222029b2: Beginning new tablet copy session on tablet 26841e9dcbca4d6d995fba9546ada63c from peer 32544f993fc24d128352e8f4f04d5eeb at {username='slave'} at 127.0.0.1:46926: session id = 32544f993fc24d128352e8f4f04d5eeb-26841e9dcbca4d6d995fba9546ada63c
I20240501 08:05:03.321246 25678 tablet_copy_client.cc:783] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Starting download of 0 data blocks...
I20240501 08:05:03.321378 25511 tablet_copy_source_session.cc:205] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:05:03.321923 25678 tablet_copy_client.cc:647] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:03.324184 25686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26841e9dcbca4d6d995fba9546ada63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:03.325367 25678 tablet_copy_client.cc:523] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:03.331527 25678 tablet_bootstrap.cc:492] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb: Bootstrap starting.
I20240501 08:05:03.333880 25686 tablet_copy_client.cc:783] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Starting download of 0 data blocks...
I20240501 08:05:03.334362 25686 tablet_copy_client.cc:647] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:03.337863 25686 tablet_copy_client.cc:523] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:03.344099 25686 tablet_bootstrap.cc:492] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb: Bootstrap starting.
I20240501 08:05:03.345764 25678 tablet_bootstrap.cc:492] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:03.346387 25678 tablet_bootstrap.cc:492] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb: Bootstrap complete.
I20240501 08:05:03.346873 25678 ts_tablet_manager.cc:1367] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.004s
I20240501 08:05:03.348862 25678 raft_consensus.cc:348] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } }
I20240501 08:05:03.349409 25678 raft_consensus.cc:729] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 32544f993fc24d128352e8f4f04d5eeb, State: Initialized, Role: LEARNER
I20240501 08:05:03.349925 25678 consensus_queue.cc:260] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } }
I20240501 08:05:03.351835 25663 heartbeater.cc:503] Master 127.20.151.62:34541 was elected leader, sending a full tablet report...
I20240501 08:05:03.354074 25678 ts_tablet_manager.cc:1398] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240501 08:05:03.355746 25437 tablet_copy_service.cc:342] P 3725d5798b684f8d9b82b754668ae5da: Request end of tablet copy session 32544f993fc24d128352e8f4f04d5eeb-f652910a32a348339c345ce75ce79be7 received from {username='slave'} at 127.0.0.1:40376
I20240501 08:05:03.356184 25437 tablet_copy_service.cc:433] P 3725d5798b684f8d9b82b754668ae5da: ending tablet copy session 32544f993fc24d128352e8f4f04d5eeb-f652910a32a348339c345ce75ce79be7 on tablet f652910a32a348339c345ce75ce79be7 with peer 32544f993fc24d128352e8f4f04d5eeb
I20240501 08:05:03.359448 25686 tablet_bootstrap.cc:492] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:03.360277 25686 tablet_bootstrap.cc:492] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb: Bootstrap complete.
I20240501 08:05:03.360900 25686 ts_tablet_manager.cc:1367] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.005s
I20240501 08:05:03.362766 25686 raft_consensus.cc:348] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } }
I20240501 08:05:03.363370 25686 raft_consensus.cc:729] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 32544f993fc24d128352e8f4f04d5eeb, State: Initialized, Role: LEARNER
I20240501 08:05:03.363938 25686 consensus_queue.cc:260] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } }
I20240501 08:05:03.365733 25686 ts_tablet_manager.cc:1398] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20240501 08:05:03.367110 25511 tablet_copy_service.cc:342] P 526d6b18efbb4af9a9a0216e222029b2: Request end of tablet copy session 32544f993fc24d128352e8f4f04d5eeb-26841e9dcbca4d6d995fba9546ada63c received from {username='slave'} at 127.0.0.1:46926
I20240501 08:05:03.367498 25511 tablet_copy_service.cc:433] P 526d6b18efbb4af9a9a0216e222029b2: ending tablet copy session 32544f993fc24d128352e8f4f04d5eeb-26841e9dcbca4d6d995fba9546ada63c on tablet 26841e9dcbca4d6d995fba9546ada63c with peer 32544f993fc24d128352e8f4f04d5eeb
I20240501 08:05:03.503024 25686 ts_tablet_manager.cc:902] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb: Initiating tablet copy from peer 526d6b18efbb4af9a9a0216e222029b2 (127.20.151.3:37755)
I20240501 08:05:03.506139 25686 tablet_copy_client.cc:308] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.3:37755
I20240501 08:05:03.507485 25511 tablet_copy_service.cc:140] P 526d6b18efbb4af9a9a0216e222029b2: Received BeginTabletCopySession request for tablet 19ccc66f74df4b3ebc1fceb2499c0308 from peer 32544f993fc24d128352e8f4f04d5eeb ({username='slave'} at 127.0.0.1:46926)
I20240501 08:05:03.507900 25511 tablet_copy_service.cc:161] P 526d6b18efbb4af9a9a0216e222029b2: Beginning new tablet copy session on tablet 19ccc66f74df4b3ebc1fceb2499c0308 from peer 32544f993fc24d128352e8f4f04d5eeb at {username='slave'} at 127.0.0.1:46926: session id = 32544f993fc24d128352e8f4f04d5eeb-19ccc66f74df4b3ebc1fceb2499c0308
I20240501 08:05:03.512789 25511 tablet_copy_source_session.cc:205] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:05:03.515209 25686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ccc66f74df4b3ebc1fceb2499c0308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:03.524896 25686 tablet_copy_client.cc:783] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Starting download of 0 data blocks...
I20240501 08:05:03.525381 25686 tablet_copy_client.cc:647] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:03.528383 25686 tablet_copy_client.cc:523] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:03.533957 25686 tablet_bootstrap.cc:492] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb: Bootstrap starting.
I20240501 08:05:03.546906 25686 tablet_bootstrap.cc:492] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:03.547482 25686 tablet_bootstrap.cc:492] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb: Bootstrap complete.
I20240501 08:05:03.547930 25686 ts_tablet_manager.cc:1367] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20240501 08:05:03.549597 25686 raft_consensus.cc:348] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } }
I20240501 08:05:03.550199 25686 raft_consensus.cc:729] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 32544f993fc24d128352e8f4f04d5eeb, State: Initialized, Role: LEARNER
I20240501 08:05:03.550714 25686 consensus_queue.cc:260] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } }
I20240501 08:05:03.552454 25686 ts_tablet_manager.cc:1398] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 08:05:03.553871 25511 tablet_copy_service.cc:342] P 526d6b18efbb4af9a9a0216e222029b2: Request end of tablet copy session 32544f993fc24d128352e8f4f04d5eeb-19ccc66f74df4b3ebc1fceb2499c0308 received from {username='slave'} at 127.0.0.1:46926
I20240501 08:05:03.554168 25511 tablet_copy_service.cc:433] P 526d6b18efbb4af9a9a0216e222029b2: ending tablet copy session 32544f993fc24d128352e8f4f04d5eeb-19ccc66f74df4b3ebc1fceb2499c0308 on tablet 19ccc66f74df4b3ebc1fceb2499c0308 with peer 32544f993fc24d128352e8f4f04d5eeb
W20240501 08:05:03.562618 25298 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver a6a4ec80f9c949ddb15d9d10afb32cc9 not available for placement
I20240501 08:05:03.709400 25638 raft_consensus.cc:1203] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:05:03.714834 25638 raft_consensus.cc:1203] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 08:05:03.751626 25466 consensus_peers.cc:485] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 -> Peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Couldn't send request to peer a6a4ec80f9c949ddb15d9d10afb32cc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:03.877246 25668 consensus_queue.cc:579] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Leader has been unable to successfully communicate with peer a6a4ec80f9c949ddb15d9d10afb32cc9 for more than 1 seconds (1.084s)
I20240501 08:05:03.896071 25427 raft_consensus.cc:1261] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 2 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 08:05:03.897441 25668 consensus_queue.cc:1035] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:05:03.900665 25668 consensus_queue.cc:579] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Leader has been unable to successfully communicate with peer a6a4ec80f9c949ddb15d9d10afb32cc9 for more than 1 seconds (1.025s)
I20240501 08:05:03.939605 25638 raft_consensus.cc:1203] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:05:04.106778 25668 raft_consensus.cc:1050] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2: attempting to promote NON_VOTER 32544f993fc24d128352e8f4f04d5eeb to VOTER
I20240501 08:05:04.109594 25668 consensus_queue.cc:237] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } }
I20240501 08:05:04.117733 25427 raft_consensus.cc:1261] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:05:04.117750 25638 raft_consensus.cc:1261] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb [term 1 LEARNER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:05:04.121129 25708 consensus_queue.cc:1035] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 08:05:04.122350 25700 consensus_queue.cc:1035] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240501 08:05:04.123255 25466 consensus_peers.cc:485] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 -> Peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Couldn't send request to peer a6a4ec80f9c949ddb15d9d10afb32cc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:04.124760 25589 raft_consensus.cc:1050] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da: attempting to promote NON_VOTER 32544f993fc24d128352e8f4f04d5eeb to VOTER
I20240501 08:05:04.129545 25589 consensus_queue.cc:237] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } }
I20240501 08:05:04.138428 25638 raft_consensus.cc:1261] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb [term 1 LEARNER]: Refusing update from remote peer 3725d5798b684f8d9b82b754668ae5da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:05:04.139407 25501 raft_consensus.cc:1261] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Refusing update from remote peer 3725d5798b684f8d9b82b754668ae5da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:05:04.139257 25637 raft_consensus.cc:2932] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
W20240501 08:05:04.141949 25392 consensus_peers.cc:485] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da -> Peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Couldn't send request to peer a6a4ec80f9c949ddb15d9d10afb32cc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:04.136339 25668 raft_consensus.cc:2932] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.146371 25702 consensus_queue.cc:1035] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [LEADER]: Connected to new peer: Peer: permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 08:05:04.147343 25701 consensus_queue.cc:1035] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [LEADER]: Connected to new peer: Peer: permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:05:04.161141 25427 raft_consensus.cc:2932] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.168264 25249 catalog_manager.cc:5484] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb reported cstate change: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "526d6b18efbb4af9a9a0216e222029b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.173300 25701 raft_consensus.cc:2932] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.175030 25501 raft_consensus.cc:2932] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.184266 25637 raft_consensus.cc:2932] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.198774 25501 consensus_queue.cc:237] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } }
I20240501 08:05:04.198293 25248 catalog_manager.cc:5484] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb reported cstate change: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3725d5798b684f8d9b82b754668ae5da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.204878 25427 raft_consensus.cc:1261] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:05:04.206132 25637 raft_consensus.cc:1261] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:05:04.206367 25668 consensus_queue.cc:1035] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:05:04.209455 25700 consensus_queue.cc:1035] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:05:04.213586 25426 consensus_queue.cc:237] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } }
I20240501 08:05:04.214056 25700 raft_consensus.cc:2932] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.220460 25637 raft_consensus.cc:1261] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Refusing update from remote peer 3725d5798b684f8d9b82b754668ae5da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:05:04.221184 25501 raft_consensus.cc:1261] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Refusing update from remote peer 3725d5798b684f8d9b82b754668ae5da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:05:04.221874 25638 raft_consensus.cc:2932] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.222748 25701 consensus_queue.cc:1035] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [LEADER]: Connected to new peer: Peer: permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:05:04.224323 25712 consensus_queue.cc:1035] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [LEADER]: Connected to new peer: Peer: permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 08:05:04.221885 25426 raft_consensus.cc:2932] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.226337 25235 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 26841e9dcbca4d6d995fba9546ada63c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:04.228592 25247 catalog_manager.cc:5484] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 reported cstate change: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New cstate: current_term: 1 leader_uuid: "526d6b18efbb4af9a9a0216e222029b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:05:04.235344 25589 raft_consensus.cc:2932] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.243276 25637 raft_consensus.cc:2932] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.252821 25236 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f652910a32a348339c345ce75ce79be7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:04.251718 25501 raft_consensus.cc:2932] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.259426 25249 catalog_manager.cc:5484] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da reported cstate change: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New cstate: current_term: 1 leader_uuid: "3725d5798b684f8d9b82b754668ae5da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:05:04.263720 25236 catalog_manager.cc:4573] TS a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26841e9dcbca4d6d995fba9546ada63c: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111)
I20240501 08:05:04.420150 25708 raft_consensus.cc:1050] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2: attempting to promote NON_VOTER 32544f993fc24d128352e8f4f04d5eeb to VOTER
I20240501 08:05:04.422036 25708 consensus_queue.cc:237] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } }
I20240501 08:05:04.427227 25637 raft_consensus.cc:1261] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb [term 1 LEARNER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:05:04.428097 25426 raft_consensus.cc:1261] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:05:04.428575 25708 consensus_queue.cc:1035] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:05:04.429600 25668 consensus_queue.cc:1035] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 08:05:04.430291 25466 consensus_peers.cc:485] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 -> Peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Couldn't send request to peer a6a4ec80f9c949ddb15d9d10afb32cc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:04.438184 25708 raft_consensus.cc:2932] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.439738 25637 raft_consensus.cc:2932] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.448020 25426 raft_consensus.cc:2932] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.449391 25248 catalog_manager.cc:5484] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 reported cstate change: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "526d6b18efbb4af9a9a0216e222029b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:05:04.462389 25501 consensus_queue.cc:237] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } }
I20240501 08:05:04.466753 25637 raft_consensus.cc:1261] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:05:04.466991 25426 raft_consensus.cc:1261] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:05:04.468161 25708 consensus_queue.cc:1035] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:05:04.468916 25700 consensus_queue.cc:1035] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:05:04.474678 25668 raft_consensus.cc:2932] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.476054 25637 raft_consensus.cc:2932] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.476332 25426 raft_consensus.cc:2932] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.484813 25235 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 19ccc66f74df4b3ebc1fceb2499c0308 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:04.485589 25247 catalog_manager.cc:5484] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb reported cstate change: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New cstate: current_term: 1 leader_uuid: "526d6b18efbb4af9a9a0216e222029b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:04.542806 25668 consensus_queue.cc:579] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Leader has been unable to successfully communicate with peer a6a4ec80f9c949ddb15d9d10afb32cc9 for more than 1 seconds (1.263s)
I20240501 08:05:04.551285 25501 consensus_queue.cc:237] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } }
I20240501 08:05:04.556797 25426 raft_consensus.cc:1261] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 2 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240501 08:05:04.557361 25464 consensus_peers.cc:485] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 -> Peer 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4:46233): Couldn't send request to peer 32544f993fc24d128352e8f4f04d5eeb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d59079b3a4f44463b337d12cf267ca54. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:04.558109 25668 consensus_queue.cc:1035] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:05:04.559379 25466 consensus_peers.cc:485] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 -> Peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Couldn't send request to peer a6a4ec80f9c949ddb15d9d10afb32cc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:04.564020 25708 raft_consensus.cc:2932] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } } }
W20240501 08:05:04.565069 25298 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver a6a4ec80f9c949ddb15d9d10afb32cc9 not available for placement
I20240501 08:05:04.565304 25426 raft_consensus.cc:2932] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } } }
I20240501 08:05:04.571669 25235 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d59079b3a4f44463b337d12cf267ca54 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:05:04.575641 25248 catalog_manager.cc:5484] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 reported cstate change: config changed from index -1 to 2, NON_VOTER 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) added. New cstate: current_term: 2 leader_uuid: "526d6b18efbb4af9a9a0216e222029b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:05.080608 25723 ts_tablet_manager.cc:902] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb: Initiating tablet copy from peer 526d6b18efbb4af9a9a0216e222029b2 (127.20.151.3:37755)
I20240501 08:05:05.083891 25723 tablet_copy_client.cc:308] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.3:37755
I20240501 08:05:05.085630 25511 tablet_copy_service.cc:140] P 526d6b18efbb4af9a9a0216e222029b2: Received BeginTabletCopySession request for tablet d59079b3a4f44463b337d12cf267ca54 from peer 32544f993fc24d128352e8f4f04d5eeb ({username='slave'} at 127.0.0.1:46926)
I20240501 08:05:05.086150 25511 tablet_copy_service.cc:161] P 526d6b18efbb4af9a9a0216e222029b2: Beginning new tablet copy session on tablet d59079b3a4f44463b337d12cf267ca54 from peer 32544f993fc24d128352e8f4f04d5eeb at {username='slave'} at 127.0.0.1:46926: session id = 32544f993fc24d128352e8f4f04d5eeb-d59079b3a4f44463b337d12cf267ca54
I20240501 08:05:05.092197 25511 tablet_copy_source_session.cc:205] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:05:05.094839 25723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59079b3a4f44463b337d12cf267ca54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:05.105917 25723 tablet_copy_client.cc:783] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Starting download of 0 data blocks...
I20240501 08:05:05.106478 25723 tablet_copy_client.cc:647] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:05.109508 25723 tablet_copy_client.cc:523] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:05.115012 25723 tablet_bootstrap.cc:492] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb: Bootstrap starting.
I20240501 08:05:05.128234 25723 tablet_bootstrap.cc:492] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:05.128888 25723 tablet_bootstrap.cc:492] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb: Bootstrap complete.
I20240501 08:05:05.129385 25723 ts_tablet_manager.cc:1367] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20240501 08:05:05.131309 25723 raft_consensus.cc:348] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } }
I20240501 08:05:05.131860 25723 raft_consensus.cc:729] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 32544f993fc24d128352e8f4f04d5eeb, State: Initialized, Role: LEARNER
I20240501 08:05:05.132364 25723 consensus_queue.cc:260] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: true } }
I20240501 08:05:05.134701 25723 ts_tablet_manager.cc:1398] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20240501 08:05:05.136137 25511 tablet_copy_service.cc:342] P 526d6b18efbb4af9a9a0216e222029b2: Request end of tablet copy session 32544f993fc24d128352e8f4f04d5eeb-d59079b3a4f44463b337d12cf267ca54 received from {username='slave'} at 127.0.0.1:46926
I20240501 08:05:05.136435 25511 tablet_copy_service.cc:433] P 526d6b18efbb4af9a9a0216e222029b2: ending tablet copy session 32544f993fc24d128352e8f4f04d5eeb-d59079b3a4f44463b337d12cf267ca54 on tablet d59079b3a4f44463b337d12cf267ca54 with peer 32544f993fc24d128352e8f4f04d5eeb
W20240501 08:05:05.406651 25236 catalog_manager.cc:4573] TS a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26841e9dcbca4d6d995fba9546ada63c: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111)
I20240501 08:05:05.547848 25637 raft_consensus.cc:1203] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
W20240501 08:05:05.567075 25298 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver a6a4ec80f9c949ddb15d9d10afb32cc9 not available for placement
I20240501 08:05:05.586088 25731 consensus_queue.cc:579] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Leader has been unable to successfully communicate with peer a6a4ec80f9c949ddb15d9d10afb32cc9 for more than 1 seconds (1.034s)
I20240501 08:05:06.053195 25731 raft_consensus.cc:1050] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2: attempting to promote NON_VOTER 32544f993fc24d128352e8f4f04d5eeb to VOTER
I20240501 08:05:06.055014 25731 consensus_queue.cc:237] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } }
I20240501 08:05:06.061144 25637 raft_consensus.cc:1261] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb [term 2 LEARNER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:05:06.061189 25426 raft_consensus.cc:1261] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 2 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:05:06.062556 25735 consensus_queue.cc:1035] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:05:06.063267 25731 consensus_queue.cc:1035] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 08:05:06.063982 25466 consensus_peers.cc:485] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 -> Peer a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): Couldn't send request to peer a6a4ec80f9c949ddb15d9d10afb32cc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:06.071890 25735 raft_consensus.cc:2932] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:06.073395 25426 raft_consensus.cc:2932] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:06.081003 25637 raft_consensus.cc:2932] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:06.082772 25248 catalog_manager.cc:5484] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 reported cstate change: config changed from index 2 to 3, 32544f993fc24d128352e8f4f04d5eeb (127.20.151.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "526d6b18efbb4af9a9a0216e222029b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6a4ec80f9c949ddb15d9d10afb32cc9" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:05:06.097898 25501 consensus_queue.cc:237] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } }
I20240501 08:05:06.102743 25637 raft_consensus.cc:1261] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb [term 2 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:05:06.102840 25426 raft_consensus.cc:1261] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 2 FOLLOWER]: Refusing update from remote peer 526d6b18efbb4af9a9a0216e222029b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:05:06.104012 25731 consensus_queue.cc:1035] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:05:06.104833 25730 consensus_queue.cc:1035] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 08:05:06.110571 25730 raft_consensus.cc:2932] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:06.112089 25426 raft_consensus.cc:2932] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:06.114291 25638 raft_consensus.cc:2932] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
I20240501 08:05:06.121346 25235 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d59079b3a4f44463b337d12cf267ca54 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:05:06.122443 25247 catalog_manager.cc:5484] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da reported cstate change: config changed from index 3 to 4, VOTER a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1) evicted. New cstate: current_term: 2 leader_uuid: "526d6b18efbb4af9a9a0216e222029b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3725d5798b684f8d9b82b754668ae5da" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43465 } } peers { permanent_uuid: "526d6b18efbb4af9a9a0216e222029b2" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37755 } } peers { permanent_uuid: "32544f993fc24d128352e8f4f04d5eeb" member_type: VOTER last_known_addr { host: "127.20.151.4" port: 46233 } attrs { promote: false } } }
W20240501 08:05:06.443048 25236 catalog_manager.cc:4573] TS a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26841e9dcbca4d6d995fba9546ada63c: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111)
W20240501 08:05:06.844398 25236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:05:08.420186 25236 catalog_manager.cc:4573] TS a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d59079b3a4f44463b337d12cf267ca54: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111)
I20240501 08:05:09.781263 21084 tablet_server.cc:178] TabletServer@127.20.151.2:0 shutting down...
I20240501 08:05:09.807658 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:05:09.808398 21084 tablet_replica.cc:330] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da: stopping tablet replica
I20240501 08:05:09.809073 21084 raft_consensus.cc:2229] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:05:09.809892 21084 raft_consensus.cc:2258] T f652910a32a348339c345ce75ce79be7 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:09.811730 21084 tablet_replica.cc:330] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da: stopping tablet replica
I20240501 08:05:09.812215 21084 raft_consensus.cc:2229] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:09.812747 21084 raft_consensus.cc:2258] T 26841e9dcbca4d6d995fba9546ada63c P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:09.814469 21084 tablet_replica.cc:330] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da: stopping tablet replica
I20240501 08:05:09.814916 21084 raft_consensus.cc:2229] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:09.815426 21084 raft_consensus.cc:2258] T 19ccc66f74df4b3ebc1fceb2499c0308 P 3725d5798b684f8d9b82b754668ae5da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:09.817049 21084 tablet_replica.cc:330] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da: stopping tablet replica
I20240501 08:05:09.817447 21084 raft_consensus.cc:2229] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:09.817984 21084 raft_consensus.cc:2258] T d59079b3a4f44463b337d12cf267ca54 P 3725d5798b684f8d9b82b754668ae5da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:09.842350 21084 tablet_server.cc:194] TabletServer@127.20.151.2:0 shutdown complete.
W20240501 08:05:09.844341 25466 consensus_peers.cc:485] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 -> Peer 3725d5798b684f8d9b82b754668ae5da (127.20.151.2:43465): Couldn't send request to peer 3725d5798b684f8d9b82b754668ae5da. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.2:43465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:09.855943 21084 tablet_server.cc:178] TabletServer@127.20.151.3:0 shutting down...
I20240501 08:05:09.880163 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:05:09.880952 21084 tablet_replica.cc:330] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2: stopping tablet replica
I20240501 08:05:09.881577 21084 raft_consensus.cc:2229] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:05:09.882341 21084 raft_consensus.cc:2258] T d59079b3a4f44463b337d12cf267ca54 P 526d6b18efbb4af9a9a0216e222029b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:09.884248 21084 tablet_replica.cc:330] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2: stopping tablet replica
I20240501 08:05:09.884717 21084 raft_consensus.cc:2229] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:05:09.885386 21084 raft_consensus.cc:2258] T 19ccc66f74df4b3ebc1fceb2499c0308 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:09.887444 21084 tablet_replica.cc:330] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2: stopping tablet replica
I20240501 08:05:09.887883 21084 raft_consensus.cc:2229] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:05:09.888633 21084 raft_consensus.cc:2258] T 26841e9dcbca4d6d995fba9546ada63c P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:09.890496 21084 tablet_replica.cc:330] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2: stopping tablet replica
I20240501 08:05:09.890944 21084 raft_consensus.cc:2229] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:09.891417 21084 raft_consensus.cc:2258] T f652910a32a348339c345ce75ce79be7 P 526d6b18efbb4af9a9a0216e222029b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:09.913933 21084 tablet_server.cc:194] TabletServer@127.20.151.3:0 shutdown complete.
I20240501 08:05:09.926694 21084 tablet_server.cc:178] TabletServer@127.20.151.4:0 shutting down...
I20240501 08:05:09.947000 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:05:09.947688 21084 tablet_replica.cc:330] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb: stopping tablet replica
I20240501 08:05:09.948223 21084 raft_consensus.cc:2229] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:09.948704 21084 raft_consensus.cc:2258] T d59079b3a4f44463b337d12cf267ca54 P 32544f993fc24d128352e8f4f04d5eeb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:09.950425 21084 tablet_replica.cc:330] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb: stopping tablet replica
I20240501 08:05:09.950919 21084 raft_consensus.cc:2229] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:09.951413 21084 raft_consensus.cc:2258] T 19ccc66f74df4b3ebc1fceb2499c0308 P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:09.953217 21084 tablet_replica.cc:330] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb: stopping tablet replica
I20240501 08:05:09.953629 21084 raft_consensus.cc:2229] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:09.954135 21084 raft_consensus.cc:2258] T 26841e9dcbca4d6d995fba9546ada63c P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:09.955775 21084 tablet_replica.cc:330] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb: stopping tablet replica
I20240501 08:05:09.956220 21084 raft_consensus.cc:2229] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:09.956635 21084 raft_consensus.cc:2258] T f652910a32a348339c345ce75ce79be7 P 32544f993fc24d128352e8f4f04d5eeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:09.978170 21084 tablet_server.cc:194] TabletServer@127.20.151.4:0 shutdown complete.
I20240501 08:05:09.990020 21084 master.cc:537] Master@127.20.151.62:34541 shutting down...
W20240501 08:05:10.476570 25236 catalog_manager.cc:4573] TS a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d59079b3a4f44463b337d12cf267ca54: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111)
W20240501 08:05:12.663311 25236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240501 08:05:12.666401 25236 catalog_manager.cc:4573] TS a6a4ec80f9c949ddb15d9d10afb32cc9 (127.20.151.1:45579): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26841e9dcbca4d6d995fba9546ada63c: Network error: Client connection negotiation failed: client connection to 127.20.151.1:45579: connect: Connection refused (error 111)
I20240501 08:05:12.888206 21084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:05:12.889482 21084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:12.889916 21084 tablet_replica.cc:330] T 00000000000000000000000000000000 P e205c2d1549b4ab492d4ff565075239b: stopping tablet replica
I20240501 08:05:12.911254 21084 master.cc:552] Master@127.20.151.62:34541 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (13527 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20240501 08:05:12.948864 21084 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.151.62:44219
I20240501 08:05:12.950027 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:05:12.955839 25755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:12.957096 25756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:05:12.958195 21084 server_base.cc:1034] running on GCE node
W20240501 08:05:12.958647 25758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:05:12.959441 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:05:12.959661 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:05:12.959811 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550712959794 us; error 0 us; skew 500 ppm
I20240501 08:05:12.960331 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:05:12.962918 21084 webserver.cc:460] Webserver started at http://127.20.151.62:41093/ using document root <none> and password file <none>
I20240501 08:05:12.963359 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:05:12.963533 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:05:12.963784 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:05:12.964838 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/master-0-root/instance:
uuid: "bd12f524deaa4caca9201b3f3194d951"
format_stamp: "Formatted at 2024-05-01 08:05:12 on dist-test-slave-pwkb"
I20240501 08:05:12.969246 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240501 08:05:12.972478 25763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:12.973294 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:05:12.973578 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/master-0-root
uuid: "bd12f524deaa4caca9201b3f3194d951"
format_stamp: "Formatted at 2024-05-01 08:05:12 on dist-test-slave-pwkb"
I20240501 08:05:12.973919 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:05:12.989569 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:05:12.990759 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:05:13.030050 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:44219
I20240501 08:05:13.030143 25814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:44219 every 8 connection(s)
I20240501 08:05:13.033856 25815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:13.044534 25815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951: Bootstrap starting.
I20240501 08:05:13.048699 25815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:13.052649 25815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951: No bootstrap required, opened a new log
I20240501 08:05:13.054569 25815 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd12f524deaa4caca9201b3f3194d951" member_type: VOTER }
I20240501 08:05:13.054931 25815 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:13.055151 25815 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd12f524deaa4caca9201b3f3194d951, State: Initialized, Role: FOLLOWER
I20240501 08:05:13.055677 25815 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd12f524deaa4caca9201b3f3194d951" member_type: VOTER }
I20240501 08:05:13.056107 25815 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:05:13.056316 25815 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:05:13.056571 25815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:13.060770 25815 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd12f524deaa4caca9201b3f3194d951" member_type: VOTER }
I20240501 08:05:13.061234 25815 leader_election.cc:302] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd12f524deaa4caca9201b3f3194d951; no voters:
I20240501 08:05:13.062300 25815 leader_election.cc:288] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:05:13.062686 25818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:05:13.063956 25818 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [term 1 LEADER]: Becoming Leader. State: Replica: bd12f524deaa4caca9201b3f3194d951, State: Running, Role: LEADER
I20240501 08:05:13.064579 25818 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd12f524deaa4caca9201b3f3194d951" member_type: VOTER }
I20240501 08:05:13.064960 25815 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:05:13.067236 25820 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd12f524deaa4caca9201b3f3194d951. Latest consensus state: current_term: 1 leader_uuid: "bd12f524deaa4caca9201b3f3194d951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd12f524deaa4caca9201b3f3194d951" member_type: VOTER } }
I20240501 08:05:13.067366 25819 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd12f524deaa4caca9201b3f3194d951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd12f524deaa4caca9201b3f3194d951" member_type: VOTER } }
I20240501 08:05:13.067867 25820 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [sys.catalog]: This master's current role is: LEADER
I20240501 08:05:13.067976 25819 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951 [sys.catalog]: This master's current role is: LEADER
I20240501 08:05:13.072160 25823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:05:13.077025 25823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:05:13.080262 21084 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:05:13.084784 25823 catalog_manager.cc:1347] Generated new cluster ID: f88d0a47601e4da9b9da101a0effccbf
I20240501 08:05:13.084990 25823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:05:13.095206 25823 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:05:13.096408 25823 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:05:13.112529 25823 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bd12f524deaa4caca9201b3f3194d951: Generated new TSK 0
I20240501 08:05:13.113104 25823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:05:13.146564 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:05:13.152475 25836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:13.154215 25837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:13.155433 25839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:05:13.156481 21084 server_base.cc:1034] running on GCE node
I20240501 08:05:13.157186 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:05:13.157377 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:05:13.157497 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550713157486 us; error 0 us; skew 500 ppm
I20240501 08:05:13.158022 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:05:13.160302 21084 webserver.cc:460] Webserver started at http://127.20.151.1:37475/ using document root <none> and password file <none>
I20240501 08:05:13.160713 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:05:13.160861 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:05:13.161063 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:05:13.162160 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-0-root/instance:
uuid: "a8bdf0eecb024a989a57ea4a16594503"
format_stamp: "Formatted at 2024-05-01 08:05:13 on dist-test-slave-pwkb"
I20240501 08:05:13.166603 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 08:05:13.169667 25844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:13.170503 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:05:13.170763 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-0-root
uuid: "a8bdf0eecb024a989a57ea4a16594503"
format_stamp: "Formatted at 2024-05-01 08:05:13 on dist-test-slave-pwkb"
I20240501 08:05:13.171039 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:05:13.190078 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:05:13.191195 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:05:13.192641 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:05:13.194990 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:05:13.195158 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:13.195395 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:05:13.195571 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:13.233919 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:40455
I20240501 08:05:13.233985 25906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:40455 every 8 connection(s)
I20240501 08:05:13.238361 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:05:13.246353 25911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:13.249311 25912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:13.252009 25914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:05:13.252264 25907 heartbeater.cc:347] Connected to a master server at 127.20.151.62:44219
I20240501 08:05:13.252365 21084 server_base.cc:1034] running on GCE node
I20240501 08:05:13.253019 25907 heartbeater.cc:464] Registering TS with master...
I20240501 08:05:13.253470 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:05:13.253813 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:05:13.253829 25907 heartbeater.cc:511] Master 127.20.151.62:44219 requested a full tablet report, sending...
I20240501 08:05:13.254038 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550713254019 us; error 0 us; skew 500 ppm
I20240501 08:05:13.254724 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:05:13.256085 25780 ts_manager.cc:194] Registered new tserver with Master: a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455)
I20240501 08:05:13.257385 21084 webserver.cc:460] Webserver started at http://127.20.151.2:32865/ using document root <none> and password file <none>
I20240501 08:05:13.257892 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:05:13.258075 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:05:13.258168 25780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48460
I20240501 08:05:13.258317 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:05:13.259701 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-1-root/instance:
uuid: "412c21db44284526856da410d118cd9a"
format_stamp: "Formatted at 2024-05-01 08:05:13 on dist-test-slave-pwkb"
I20240501 08:05:13.264163 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:05:13.267665 25919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:13.268479 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 08:05:13.268736 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-1-root
uuid: "412c21db44284526856da410d118cd9a"
format_stamp: "Formatted at 2024-05-01 08:05:13 on dist-test-slave-pwkb"
I20240501 08:05:13.268980 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:05:13.312006 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:05:13.313400 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:05:13.314829 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:05:13.317004 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:05:13.317183 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:13.317423 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:05:13.317585 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:13.357138 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:39107
I20240501 08:05:13.357223 25981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:39107 every 8 connection(s)
I20240501 08:05:13.361972 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:05:13.368973 25985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:13.370429 25986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:13.375358 25988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:05:13.376255 25982 heartbeater.cc:347] Connected to a master server at 127.20.151.62:44219
I20240501 08:05:13.376634 25982 heartbeater.cc:464] Registering TS with master...
I20240501 08:05:13.377311 25982 heartbeater.cc:511] Master 127.20.151.62:44219 requested a full tablet report, sending...
I20240501 08:05:13.378100 21084 server_base.cc:1034] running on GCE node
I20240501 08:05:13.378978 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:05:13.379171 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:05:13.379315 25780 ts_manager.cc:194] Registered new tserver with Master: 412c21db44284526856da410d118cd9a (127.20.151.2:39107)
I20240501 08:05:13.379348 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550713379328 us; error 0 us; skew 500 ppm
I20240501 08:05:13.380147 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:05:13.380897 25780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48472
I20240501 08:05:13.383307 21084 webserver.cc:460] Webserver started at http://127.20.151.3:41153/ using document root <none> and password file <none>
I20240501 08:05:13.383754 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:05:13.383893 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:05:13.384084 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:05:13.385110 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-2-root/instance:
uuid: "f1b6af9bbeab4df9a81e15a2965e6650"
format_stamp: "Formatted at 2024-05-01 08:05:13 on dist-test-slave-pwkb"
I20240501 08:05:13.389632 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:05:13.392802 25993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:13.393622 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:05:13.393916 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-2-root
uuid: "f1b6af9bbeab4df9a81e15a2965e6650"
format_stamp: "Formatted at 2024-05-01 08:05:13 on dist-test-slave-pwkb"
I20240501 08:05:13.394163 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:05:13.405959 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:05:13.406992 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:05:13.408362 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:05:13.410609 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:05:13.410778 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:13.411003 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:05:13.411157 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:13.450634 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:45489
I20240501 08:05:13.450714 26055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:45489 every 8 connection(s)
I20240501 08:05:13.464021 26056 heartbeater.cc:347] Connected to a master server at 127.20.151.62:44219
I20240501 08:05:13.464365 26056 heartbeater.cc:464] Registering TS with master...
I20240501 08:05:13.465081 26056 heartbeater.cc:511] Master 127.20.151.62:44219 requested a full tablet report, sending...
I20240501 08:05:13.466899 25780 ts_manager.cc:194] Registered new tserver with Master: f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:13.467275 21084 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01358283s
I20240501 08:05:13.468150 25780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48486
I20240501 08:05:14.260998 25907 heartbeater.cc:503] Master 127.20.151.62:44219 was elected leader, sending a full tablet report...
I20240501 08:05:14.383843 25982 heartbeater.cc:503] Master 127.20.151.62:44219 was elected leader, sending a full tablet report...
I20240501 08:05:14.470993 26056 heartbeater.cc:503] Master 127.20.151.62:44219 was elected leader, sending a full tablet report...
I20240501 08:05:14.499787 21084 test_util.cc:268] Using random seed: -230063508
I20240501 08:05:14.529348 25780 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48502:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:05:14.531929 25780 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:05:14.587990 25947 tablet_service.cc:1467] Processing CreateTablet for tablet b049db705b3242ecbd1b6483bfee9813 (DEFAULT_TABLE table=test-workload [id=d82f50f2bb23411794d026356c93a9ef]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 08:05:14.588857 25944 tablet_service.cc:1467] Processing CreateTablet for tablet c22bf70d91434f688f21d7a288a2d355 (DEFAULT_TABLE table=test-workload [id=d82f50f2bb23411794d026356c93a9ef]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 08:05:14.590539 25947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b049db705b3242ecbd1b6483bfee9813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:14.591432 25945 tablet_service.cc:1467] Processing CreateTablet for tablet d8aa10ca2f384362841703c074e8ea3b (DEFAULT_TABLE table=test-workload [id=d82f50f2bb23411794d026356c93a9ef]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 08:05:14.592815 25945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8aa10ca2f384362841703c074e8ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:14.594957 25946 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9e3f16ca2242069311a4aa01e1a9d3 (DEFAULT_TABLE table=test-workload [id=d82f50f2bb23411794d026356c93a9ef]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 08:05:14.597290 25946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9e3f16ca2242069311a4aa01e1a9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:14.600884 25944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22bf70d91434f688f21d7a288a2d355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:14.619632 25872 tablet_service.cc:1467] Processing CreateTablet for tablet b049db705b3242ecbd1b6483bfee9813 (DEFAULT_TABLE table=test-workload [id=d82f50f2bb23411794d026356c93a9ef]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 08:05:14.621165 25872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b049db705b3242ecbd1b6483bfee9813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:14.624289 26076 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a: Bootstrap starting.
I20240501 08:05:14.627112 25871 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9e3f16ca2242069311a4aa01e1a9d3 (DEFAULT_TABLE table=test-workload [id=d82f50f2bb23411794d026356c93a9ef]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 08:05:14.628516 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9e3f16ca2242069311a4aa01e1a9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:14.628475 26021 tablet_service.cc:1467] Processing CreateTablet for tablet b049db705b3242ecbd1b6483bfee9813 (DEFAULT_TABLE table=test-workload [id=d82f50f2bb23411794d026356c93a9ef]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 08:05:14.630916 26020 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9e3f16ca2242069311a4aa01e1a9d3 (DEFAULT_TABLE table=test-workload [id=d82f50f2bb23411794d026356c93a9ef]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 08:05:14.632220 25869 tablet_service.cc:1467] Processing CreateTablet for tablet c22bf70d91434f688f21d7a288a2d355 (DEFAULT_TABLE table=test-workload [id=d82f50f2bb23411794d026356c93a9ef]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 08:05:14.633198 26019 tablet_service.cc:1467] Processing CreateTablet for tablet d8aa10ca2f384362841703c074e8ea3b (DEFAULT_TABLE table=test-workload [id=d82f50f2bb23411794d026356c93a9ef]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 08:05:14.630951 25870 tablet_service.cc:1467] Processing CreateTablet for tablet d8aa10ca2f384362841703c074e8ea3b (DEFAULT_TABLE table=test-workload [id=d82f50f2bb23411794d026356c93a9ef]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 08:05:14.635579 25869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22bf70d91434f688f21d7a288a2d355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:14.635458 26018 tablet_service.cc:1467] Processing CreateTablet for tablet c22bf70d91434f688f21d7a288a2d355 (DEFAULT_TABLE table=test-workload [id=d82f50f2bb23411794d026356c93a9ef]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 08:05:14.637025 26021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b049db705b3242ecbd1b6483bfee9813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:14.638247 26018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22bf70d91434f688f21d7a288a2d355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:14.642055 26020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9e3f16ca2242069311a4aa01e1a9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:14.642076 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8aa10ca2f384362841703c074e8ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:14.646063 26019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8aa10ca2f384362841703c074e8ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:14.667616 26077 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503: Bootstrap starting.
I20240501 08:05:14.678186 26077 tablet_bootstrap.cc:654] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:14.679178 26076 tablet_bootstrap.cc:654] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:14.684033 26078 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap starting.
I20240501 08:05:14.685940 26077 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503: No bootstrap required, opened a new log
I20240501 08:05:14.686569 26077 ts_tablet_manager.cc:1367] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
I20240501 08:05:14.687899 26076 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a: No bootstrap required, opened a new log
I20240501 08:05:14.688280 26076 ts_tablet_manager.cc:1367] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a: Time spent bootstrapping tablet: real 0.064s user 0.010s sys 0.004s
I20240501 08:05:14.689653 26077 raft_consensus.cc:348] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.690317 26076 raft_consensus.cc:348] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.690759 26077 raft_consensus.cc:374] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:14.690953 26076 raft_consensus.cc:374] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:14.690986 26078 tablet_bootstrap.cc:654] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:14.691179 26077 raft_consensus.cc:729] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8bdf0eecb024a989a57ea4a16594503, State: Initialized, Role: FOLLOWER
I20240501 08:05:14.692102 26077 consensus_queue.cc:260] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.691368 26076 raft_consensus.cc:729] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412c21db44284526856da410d118cd9a, State: Initialized, Role: FOLLOWER
I20240501 08:05:14.693521 26076 consensus_queue.cc:260] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.695623 26077 ts_tablet_manager.cc:1398] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240501 08:05:14.698374 26076 ts_tablet_manager.cc:1398] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20240501 08:05:14.699406 26076 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: Bootstrap starting.
I20240501 08:05:14.703480 26078 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650: No bootstrap required, opened a new log
I20240501 08:05:14.704017 26078 ts_tablet_manager.cc:1367] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.005s
I20240501 08:05:14.705559 26076 tablet_bootstrap.cc:654] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:14.698556 26077 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: Bootstrap starting.
I20240501 08:05:14.706902 26078 raft_consensus.cc:348] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.709220 26078 raft_consensus.cc:374] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:14.709599 26078 raft_consensus.cc:729] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b6af9bbeab4df9a81e15a2965e6650, State: Initialized, Role: FOLLOWER
I20240501 08:05:14.710429 26078 consensus_queue.cc:260] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.711233 26076 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: No bootstrap required, opened a new log
I20240501 08:05:14.711747 26076 ts_tablet_manager.cc:1367] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240501 08:05:14.713601 26077 tablet_bootstrap.cc:654] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:14.715189 26076 raft_consensus.cc:348] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.715997 26076 raft_consensus.cc:374] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:14.716296 26076 raft_consensus.cc:729] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412c21db44284526856da410d118cd9a, State: Initialized, Role: FOLLOWER
I20240501 08:05:14.716490 26078 ts_tablet_manager.cc:1398] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650: Time spent starting tablet: real 0.012s user 0.002s sys 0.007s
I20240501 08:05:14.717433 26078 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap starting.
I20240501 08:05:14.717032 26076 consensus_queue.cc:260] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.719461 26076 ts_tablet_manager.cc:1398] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240501 08:05:14.723747 26078 tablet_bootstrap.cc:654] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:14.724061 26076 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a: Bootstrap starting.
I20240501 08:05:14.725230 26077 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: No bootstrap required, opened a new log
I20240501 08:05:14.725919 26077 ts_tablet_manager.cc:1367] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.007s
I20240501 08:05:14.728638 26077 raft_consensus.cc:348] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.729554 26077 raft_consensus.cc:374] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:14.729900 26077 raft_consensus.cc:729] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8bdf0eecb024a989a57ea4a16594503, State: Initialized, Role: FOLLOWER
I20240501 08:05:14.730667 26077 consensus_queue.cc:260] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.731403 26076 tablet_bootstrap.cc:654] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:14.736594 26077 ts_tablet_manager.cc:1398] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240501 08:05:14.737604 26077 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503: Bootstrap starting.
I20240501 08:05:14.743300 26077 tablet_bootstrap.cc:654] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:14.754691 26078 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: No bootstrap required, opened a new log
I20240501 08:05:14.754993 26077 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503: No bootstrap required, opened a new log
I20240501 08:05:14.755111 26078 ts_tablet_manager.cc:1367] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Time spent bootstrapping tablet: real 0.038s user 0.027s sys 0.007s
I20240501 08:05:14.755126 26076 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a: No bootstrap required, opened a new log
I20240501 08:05:14.755563 26077 ts_tablet_manager.cc:1367] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240501 08:05:14.755774 26076 ts_tablet_manager.cc:1367] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.013s
I20240501 08:05:14.757416 26078 raft_consensus.cc:348] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.758121 26078 raft_consensus.cc:374] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:14.758365 26078 raft_consensus.cc:729] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b6af9bbeab4df9a81e15a2965e6650, State: Initialized, Role: FOLLOWER
I20240501 08:05:14.758265 26077 raft_consensus.cc:348] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.758590 26076 raft_consensus.cc:348] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.759119 26077 raft_consensus.cc:374] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:14.759312 26076 raft_consensus.cc:374] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:14.759101 26078 consensus_queue.cc:260] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.759683 26077 raft_consensus.cc:729] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8bdf0eecb024a989a57ea4a16594503, State: Initialized, Role: FOLLOWER
I20240501 08:05:14.759727 26076 raft_consensus.cc:729] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412c21db44284526856da410d118cd9a, State: Initialized, Role: FOLLOWER
I20240501 08:05:14.760596 26077 consensus_queue.cc:260] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.760756 26076 consensus_queue.cc:260] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.762943 26076 ts_tablet_manager.cc:1398] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:05:14.761423 26078 ts_tablet_manager.cc:1398] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240501 08:05:14.763908 26076 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a: Bootstrap starting.
I20240501 08:05:14.764068 26078 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap starting.
I20240501 08:05:14.764542 26077 ts_tablet_manager.cc:1398] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240501 08:05:14.765614 26077 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503: Bootstrap starting.
I20240501 08:05:14.770341 26078 tablet_bootstrap.cc:654] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:14.770542 26076 tablet_bootstrap.cc:654] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:14.771689 26077 tablet_bootstrap.cc:654] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:14.782935 26076 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a: No bootstrap required, opened a new log
I20240501 08:05:14.783548 26076 ts_tablet_manager.cc:1367] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.010s
I20240501 08:05:14.783488 26077 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503: No bootstrap required, opened a new log
I20240501 08:05:14.784305 26077 ts_tablet_manager.cc:1367] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20240501 08:05:14.786504 26076 raft_consensus.cc:348] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.787427 26076 raft_consensus.cc:374] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:14.786973 26077 raft_consensus.cc:348] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.787775 26076 raft_consensus.cc:729] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412c21db44284526856da410d118cd9a, State: Initialized, Role: FOLLOWER
I20240501 08:05:14.787950 26077 raft_consensus.cc:374] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:14.788347 26077 raft_consensus.cc:729] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8bdf0eecb024a989a57ea4a16594503, State: Initialized, Role: FOLLOWER
I20240501 08:05:14.788681 26076 consensus_queue.cc:260] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.789163 26077 consensus_queue.cc:260] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.791085 26076 ts_tablet_manager.cc:1398] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20240501 08:05:14.792007 26077 ts_tablet_manager.cc:1398] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:05:14.799024 26078 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: No bootstrap required, opened a new log
I20240501 08:05:14.799619 26078 ts_tablet_manager.cc:1367] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.022s
I20240501 08:05:14.804303 26078 raft_consensus.cc:348] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.805163 26078 raft_consensus.cc:374] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:14.805502 26078 raft_consensus.cc:729] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b6af9bbeab4df9a81e15a2965e6650, State: Initialized, Role: FOLLOWER
I20240501 08:05:14.806238 26078 consensus_queue.cc:260] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.808045 26078 ts_tablet_manager.cc:1398] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Time spent starting tablet: real 0.008s user 0.003s sys 0.001s
I20240501 08:05:14.808948 26078 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap starting.
I20240501 08:05:14.814471 26078 tablet_bootstrap.cc:654] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650: Neither blocks nor log segments found. Creating new log.
I20240501 08:05:14.819069 26078 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650: No bootstrap required, opened a new log
I20240501 08:05:14.819411 26078 ts_tablet_manager.cc:1367] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240501 08:05:14.821218 26078 raft_consensus.cc:348] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.821736 26078 raft_consensus.cc:374] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:05:14.821938 26078 raft_consensus.cc:729] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b6af9bbeab4df9a81e15a2965e6650, State: Initialized, Role: FOLLOWER
I20240501 08:05:14.822412 26078 consensus_queue.cc:260] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.823953 26078 ts_tablet_manager.cc:1398] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 08:05:14.852279 26081 raft_consensus.cc:482] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:14.852778 26081 raft_consensus.cc:504] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.854751 26081 leader_election.cc:288] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412c21db44284526856da410d118cd9a (127.20.151.2:39107), f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:14.866673 25957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aa10ca2f384362841703c074e8ea3b" candidate_uuid: "a8bdf0eecb024a989a57ea4a16594503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412c21db44284526856da410d118cd9a" is_pre_election: true
I20240501 08:05:14.866788 26031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aa10ca2f384362841703c074e8ea3b" candidate_uuid: "a8bdf0eecb024a989a57ea4a16594503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" is_pre_election: true
I20240501 08:05:14.867489 25957 raft_consensus.cc:2454] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8bdf0eecb024a989a57ea4a16594503 in term 0.
I20240501 08:05:14.867561 26031 raft_consensus.cc:2454] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8bdf0eecb024a989a57ea4a16594503 in term 0.
I20240501 08:05:14.868849 25847 leader_election.cc:302] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c21db44284526856da410d118cd9a, a8bdf0eecb024a989a57ea4a16594503; no voters:
I20240501 08:05:14.869560 26081 raft_consensus.cc:2781] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:05:14.869930 26081 raft_consensus.cc:482] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:14.870201 26081 raft_consensus.cc:3037] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:14.875536 26081 raft_consensus.cc:504] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.876981 26081 leader_election.cc:288] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [CANDIDATE]: Term 1 election: Requested vote from peers 412c21db44284526856da410d118cd9a (127.20.151.2:39107), f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:14.877859 25957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aa10ca2f384362841703c074e8ea3b" candidate_uuid: "a8bdf0eecb024a989a57ea4a16594503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412c21db44284526856da410d118cd9a"
I20240501 08:05:14.877982 26031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aa10ca2f384362841703c074e8ea3b" candidate_uuid: "a8bdf0eecb024a989a57ea4a16594503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1b6af9bbeab4df9a81e15a2965e6650"
I20240501 08:05:14.878336 25957 raft_consensus.cc:3037] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:14.878491 26031 raft_consensus.cc:3037] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:14.883088 25957 raft_consensus.cc:2454] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8bdf0eecb024a989a57ea4a16594503 in term 1.
I20240501 08:05:14.883208 26031 raft_consensus.cc:2454] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8bdf0eecb024a989a57ea4a16594503 in term 1.
I20240501 08:05:14.883987 25847 leader_election.cc:302] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c21db44284526856da410d118cd9a, a8bdf0eecb024a989a57ea4a16594503; no voters:
I20240501 08:05:14.884675 26081 raft_consensus.cc:2781] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:05:14.885779 26081 raft_consensus.cc:686] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 LEADER]: Becoming Leader. State: Replica: a8bdf0eecb024a989a57ea4a16594503, State: Running, Role: LEADER
I20240501 08:05:14.886598 26081 consensus_queue.cc:237] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.892930 25778 catalog_manager.cc:5484] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "a8bdf0eecb024a989a57ea4a16594503" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:14.908130 26085 raft_consensus.cc:482] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:14.908577 26085 raft_consensus.cc:504] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.910864 26085 leader_election.cc:288] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455), 412c21db44284526856da410d118cd9a (127.20.151.2:39107)
I20240501 08:05:14.920619 25957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9e3f16ca2242069311a4aa01e1a9d3" candidate_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412c21db44284526856da410d118cd9a" is_pre_election: true
I20240501 08:05:14.920658 25882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9e3f16ca2242069311a4aa01e1a9d3" candidate_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8bdf0eecb024a989a57ea4a16594503" is_pre_election: true
I20240501 08:05:14.921259 25957 raft_consensus.cc:2454] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1b6af9bbeab4df9a81e15a2965e6650 in term 0.
I20240501 08:05:14.921334 25882 raft_consensus.cc:2454] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1b6af9bbeab4df9a81e15a2965e6650 in term 0.
I20240501 08:05:14.922307 25996 leader_election.cc:302] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c21db44284526856da410d118cd9a, f1b6af9bbeab4df9a81e15a2965e6650; no voters:
I20240501 08:05:14.922960 26085 raft_consensus.cc:2781] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:05:14.923221 26085 raft_consensus.cc:482] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:14.923460 26085 raft_consensus.cc:3037] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:14.928273 26085 raft_consensus.cc:504] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.929965 26085 leader_election.cc:288] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [CANDIDATE]: Term 1 election: Requested vote from peers a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455), 412c21db44284526856da410d118cd9a (127.20.151.2:39107)
I20240501 08:05:14.930778 25882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9e3f16ca2242069311a4aa01e1a9d3" candidate_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8bdf0eecb024a989a57ea4a16594503"
I20240501 08:05:14.930946 25957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9e3f16ca2242069311a4aa01e1a9d3" candidate_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412c21db44284526856da410d118cd9a"
I20240501 08:05:14.931259 25882 raft_consensus.cc:3037] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:14.931453 25957 raft_consensus.cc:3037] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:14.935889 25957 raft_consensus.cc:2454] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1b6af9bbeab4df9a81e15a2965e6650 in term 1.
I20240501 08:05:14.935896 25882 raft_consensus.cc:2454] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1b6af9bbeab4df9a81e15a2965e6650 in term 1.
I20240501 08:05:14.936877 25996 leader_election.cc:302] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c21db44284526856da410d118cd9a, f1b6af9bbeab4df9a81e15a2965e6650; no voters:
I20240501 08:05:14.937599 26085 raft_consensus.cc:2781] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:05:14.938719 26085 raft_consensus.cc:686] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 LEADER]: Becoming Leader. State: Replica: f1b6af9bbeab4df9a81e15a2965e6650, State: Running, Role: LEADER
I20240501 08:05:14.939458 26085 consensus_queue.cc:237] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:14.946565 25778 catalog_manager.cc:5484] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } health_report { overall_health: HEALTHY } } }
I20240501 08:05:15.000502 26083 raft_consensus.cc:482] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:15.001010 26083 raft_consensus.cc:504] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:15.003206 26083 leader_election.cc:288] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455), f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:15.013036 25882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b049db705b3242ecbd1b6483bfee9813" candidate_uuid: "412c21db44284526856da410d118cd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8bdf0eecb024a989a57ea4a16594503" is_pre_election: true
I20240501 08:05:15.013749 25882 raft_consensus.cc:2454] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412c21db44284526856da410d118cd9a in term 0.
I20240501 08:05:15.014362 26031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b049db705b3242ecbd1b6483bfee9813" candidate_uuid: "412c21db44284526856da410d118cd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" is_pre_election: true
I20240501 08:05:15.014642 25920 leader_election.cc:302] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c21db44284526856da410d118cd9a, a8bdf0eecb024a989a57ea4a16594503; no voters:
I20240501 08:05:15.014997 26031 raft_consensus.cc:2454] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412c21db44284526856da410d118cd9a in term 0.
I20240501 08:05:15.015431 26083 raft_consensus.cc:2781] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:05:15.015756 26083 raft_consensus.cc:482] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:15.016042 26083 raft_consensus.cc:3037] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:15.020727 26083 raft_consensus.cc:504] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:15.022387 26083 leader_election.cc:288] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [CANDIDATE]: Term 1 election: Requested vote from peers a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455), f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:15.023350 25882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b049db705b3242ecbd1b6483bfee9813" candidate_uuid: "412c21db44284526856da410d118cd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8bdf0eecb024a989a57ea4a16594503"
I20240501 08:05:15.023694 26031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b049db705b3242ecbd1b6483bfee9813" candidate_uuid: "412c21db44284526856da410d118cd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1b6af9bbeab4df9a81e15a2965e6650"
I20240501 08:05:15.023998 25882 raft_consensus.cc:3037] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:15.024266 26031 raft_consensus.cc:3037] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:15.027711 26085 raft_consensus.cc:482] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:15.028085 26085 raft_consensus.cc:504] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:15.030588 25882 raft_consensus.cc:2454] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412c21db44284526856da410d118cd9a in term 1.
I20240501 08:05:15.030674 26085 leader_election.cc:288] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455), 412c21db44284526856da410d118cd9a (127.20.151.2:39107)
I20240501 08:05:15.030684 25957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22bf70d91434f688f21d7a288a2d355" candidate_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412c21db44284526856da410d118cd9a" is_pre_election: true
I20240501 08:05:15.030582 25881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22bf70d91434f688f21d7a288a2d355" candidate_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8bdf0eecb024a989a57ea4a16594503" is_pre_election: true
I20240501 08:05:15.030588 26031 raft_consensus.cc:2454] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412c21db44284526856da410d118cd9a in term 1.
I20240501 08:05:15.031756 25957 raft_consensus.cc:2454] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1b6af9bbeab4df9a81e15a2965e6650 in term 0.
I20240501 08:05:15.031800 25881 raft_consensus.cc:2454] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1b6af9bbeab4df9a81e15a2965e6650 in term 0.
I20240501 08:05:15.032119 25920 leader_election.cc:302] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c21db44284526856da410d118cd9a, a8bdf0eecb024a989a57ea4a16594503; no voters:
I20240501 08:05:15.032976 26083 raft_consensus.cc:2781] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:05:15.033051 25994 leader_election.cc:302] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8bdf0eecb024a989a57ea4a16594503, f1b6af9bbeab4df9a81e15a2965e6650; no voters:
I20240501 08:05:15.033856 26085 raft_consensus.cc:2781] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:05:15.034094 26083 raft_consensus.cc:686] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 LEADER]: Becoming Leader. State: Replica: 412c21db44284526856da410d118cd9a, State: Running, Role: LEADER
I20240501 08:05:15.034261 26085 raft_consensus.cc:482] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:15.034732 26085 raft_consensus.cc:3037] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:15.035008 26083 consensus_queue.cc:237] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:15.040208 26085 raft_consensus.cc:504] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:15.042209 26085 leader_election.cc:288] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [CANDIDATE]: Term 1 election: Requested vote from peers a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455), 412c21db44284526856da410d118cd9a (127.20.151.2:39107)
I20240501 08:05:15.042469 25881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22bf70d91434f688f21d7a288a2d355" candidate_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8bdf0eecb024a989a57ea4a16594503"
I20240501 08:05:15.042829 25957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22bf70d91434f688f21d7a288a2d355" candidate_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412c21db44284526856da410d118cd9a"
I20240501 08:05:15.043110 25881 raft_consensus.cc:3037] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:15.043424 25957 raft_consensus.cc:3037] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:05:15.043200 25778 catalog_manager.cc:5484] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 412c21db44284526856da410d118cd9a (127.20.151.2). New cstate: current_term: 1 leader_uuid: "412c21db44284526856da410d118cd9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:15.050009 25957 raft_consensus.cc:2454] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1b6af9bbeab4df9a81e15a2965e6650 in term 1.
I20240501 08:05:15.050009 25881 raft_consensus.cc:2454] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1b6af9bbeab4df9a81e15a2965e6650 in term 1.
I20240501 08:05:15.051266 25996 leader_election.cc:302] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c21db44284526856da410d118cd9a, f1b6af9bbeab4df9a81e15a2965e6650; no voters:
I20240501 08:05:15.051921 26085 raft_consensus.cc:2781] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:05:15.052388 26085 raft_consensus.cc:686] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 LEADER]: Becoming Leader. State: Replica: f1b6af9bbeab4df9a81e15a2965e6650, State: Running, Role: LEADER
I20240501 08:05:15.052978 26085 consensus_queue.cc:237] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:15.059895 25778 catalog_manager.cc:5484] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } health_report { overall_health: HEALTHY } } }
I20240501 08:05:15.072314 21084 tablet_server.cc:178] TabletServer@127.20.151.1:0 shutting down...
I20240501 08:05:15.098351 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:05:15.099126 21084 tablet_replica.cc:330] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503: stopping tablet replica
I20240501 08:05:15.099627 21084 raft_consensus.cc:2229] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:15.099967 21084 raft_consensus.cc:2258] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:15.101799 21084 tablet_replica.cc:330] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503: stopping tablet replica
I20240501 08:05:15.102242 21084 raft_consensus.cc:2229] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:15.102537 21084 raft_consensus.cc:2258] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:15.104216 21084 tablet_replica.cc:330] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: stopping tablet replica
I20240501 08:05:15.104633 21084 raft_consensus.cc:2229] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:05:15.105115 21084 pending_rounds.cc:62] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: Trying to abort 1 pending ops.
I20240501 08:05:15.105271 21084 pending_rounds.cc:68] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022799728181727232 op_type: NO_OP noop_request { }
I20240501 08:05:15.105587 21084 raft_consensus.cc:2866] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 08:05:15.105855 21084 raft_consensus.cc:2258] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:15.107744 21084 tablet_replica.cc:330] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503: stopping tablet replica
I20240501 08:05:15.108143 21084 raft_consensus.cc:2229] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:15.108419 21084 raft_consensus.cc:2258] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:15.130390 21084 tablet_server.cc:194] TabletServer@127.20.151.1:0 shutdown complete.
I20240501 08:05:15.141551 21084 tablet_server.cc:178] TabletServer@127.20.151.2:0 shutting down...
I20240501 08:05:15.162904 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:05:15.163839 21084 tablet_replica.cc:330] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a: stopping tablet replica
I20240501 08:05:15.164368 21084 raft_consensus.cc:2229] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:15.164716 21084 raft_consensus.cc:2258] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:15.166640 21084 tablet_replica.cc:330] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a: stopping tablet replica
I20240501 08:05:15.167101 21084 raft_consensus.cc:2229] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:15.167423 21084 raft_consensus.cc:2258] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:15.169078 21084 tablet_replica.cc:330] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: stopping tablet replica
I20240501 08:05:15.169836 21084 raft_consensus.cc:2229] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:05:15.170377 21084 pending_rounds.cc:62] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: Trying to abort 1 pending ops.
I20240501 08:05:15.170570 21084 pending_rounds.cc:68] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022799728791666688 op_type: NO_OP noop_request { }
I20240501 08:05:15.170867 21084 raft_consensus.cc:2866] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 08:05:15.171093 21084 raft_consensus.cc:2258] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:15.172935 21084 tablet_replica.cc:330] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a: stopping tablet replica
I20240501 08:05:15.173338 21084 raft_consensus.cc:2229] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:15.173888 21084 raft_consensus.cc:2258] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:15.195461 21084 tablet_server.cc:194] TabletServer@127.20.151.2:0 shutdown complete.
I20240501 08:05:15.206444 21084 tablet_server.cc:178] TabletServer@127.20.151.3:0 shutting down...
I20240501 08:05:15.227319 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:05:15.228140 21084 tablet_replica.cc:330] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: stopping tablet replica
I20240501 08:05:15.228695 21084 raft_consensus.cc:2229] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:05:15.229236 21084 pending_rounds.cc:62] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Trying to abort 1 pending ops.
I20240501 08:05:15.229418 21084 pending_rounds.cc:68] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022799728863911936 op_type: NO_OP noop_request { }
I20240501 08:05:15.229714 21084 raft_consensus.cc:2866] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 08:05:15.229974 21084 raft_consensus.cc:2258] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:15.232030 21084 tablet_replica.cc:330] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650: stopping tablet replica
I20240501 08:05:15.232486 21084 raft_consensus.cc:2229] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:15.232825 21084 raft_consensus.cc:2258] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:15.234475 21084 tablet_replica.cc:330] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: stopping tablet replica
I20240501 08:05:15.234926 21084 raft_consensus.cc:2229] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:05:15.235420 21084 pending_rounds.cc:62] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Trying to abort 1 pending ops.
I20240501 08:05:15.235596 21084 pending_rounds.cc:68] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022799728399245312 op_type: NO_OP noop_request { }
I20240501 08:05:15.235854 21084 raft_consensus.cc:2866] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 08:05:15.236085 21084 raft_consensus.cc:2258] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:15.237987 21084 tablet_replica.cc:330] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650: stopping tablet replica
I20240501 08:05:15.238400 21084 raft_consensus.cc:2229] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:15.238700 21084 raft_consensus.cc:2258] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:15.259558 21084 tablet_server.cc:194] TabletServer@127.20.151.3:0 shutdown complete.
I20240501 08:05:15.271271 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:05:15.278131 26107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:15.278996 26108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:15.279431 26110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:05:15.280023 21084 server_base.cc:1034] running on GCE node
I20240501 08:05:15.281128 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:05:15.281420 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:05:15.281661 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550715281645 us; error 0 us; skew 500 ppm
I20240501 08:05:15.282310 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:05:15.284837 21084 webserver.cc:460] Webserver started at http://127.20.151.4:35235/ using document root <none> and password file <none>
I20240501 08:05:15.285360 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:05:15.285635 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:05:15.286044 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:05:15.287204 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-3-root/instance:
uuid: "594bfa72576e4461aa048eb7474768fa"
format_stamp: "Formatted at 2024-05-01 08:05:15 on dist-test-slave-pwkb"
I20240501 08:05:15.291625 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:05:15.294802 26115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:15.295718 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:05:15.296092 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-3-root
uuid: "594bfa72576e4461aa048eb7474768fa"
format_stamp: "Formatted at 2024-05-01 08:05:15 on dist-test-slave-pwkb"
I20240501 08:05:15.296455 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:05:15.318397 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:05:15.319528 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:05:15.320986 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:05:15.323284 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:05:15.323593 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:15.323900 21084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:05:15.324162 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:15.361660 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.4:40155
I20240501 08:05:15.361788 26177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.4:40155 every 8 connection(s)
I20240501 08:05:15.375437 26178 heartbeater.cc:347] Connected to a master server at 127.20.151.62:44219
I20240501 08:05:15.375878 26178 heartbeater.cc:464] Registering TS with master...
I20240501 08:05:15.376612 26178 heartbeater.cc:511] Master 127.20.151.62:44219 requested a full tablet report, sending...
I20240501 08:05:15.378451 25778 ts_manager.cc:194] Registered new tserver with Master: 594bfa72576e4461aa048eb7474768fa (127.20.151.4:40155)
I20240501 08:05:15.379860 25778 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48510
W20240501 08:05:16.097508 25828 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.20.151.2:39107: connect: Connection refused (error 111)
W20240501 08:05:16.103541 25828 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.20.151.2:39107: connect: Connection refused (error 111)
W20240501 08:05:16.103919 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.20.151.2:39107: connect: Connection refused (error 111)
W20240501 08:05:16.108042 25828 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.20.151.3:45489: connect: Connection refused (error 111)
W20240501 08:05:16.108383 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.20.151.3:45489: connect: Connection refused (error 111)
W20240501 08:05:16.112675 25828 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.20.151.1:40455: connect: Connection refused (error 111)
W20240501 08:05:16.113004 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:40455: connect: Connection refused (error 111)
I20240501 08:05:16.382220 26178 heartbeater.cc:503] Master 127.20.151.62:44219 was elected leader, sending a full tablet report...
W20240501 08:05:17.143296 25828 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.20.151.3:45489: connect: Connection refused (error 111)
W20240501 08:05:17.148947 25828 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.20.151.3:45489: connect: Connection refused (error 111)
W20240501 08:05:17.149204 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.20.151.3:45489: connect: Connection refused (error 111)
W20240501 08:05:17.153117 25828 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.20.151.3:45489: connect: Connection refused (error 111)
W20240501 08:05:17.153358 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.20.151.3:45489: connect: Connection refused (error 111)
W20240501 08:05:17.157224 25828 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.20.151.1:40455: connect: Connection refused (error 111)
W20240501 08:05:17.157455 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:40455: connect: Connection refused (error 111)
W20240501 08:05:18.168684 25828 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.20.151.1:40455: connect: Connection refused (error 111)
W20240501 08:05:18.171574 25764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.1:40455: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 08:05:18.174573 25828 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.20.151.1:40455: connect: Connection refused (error 111)
W20240501 08:05:18.174834 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:40455: connect: Connection refused (error 111)
W20240501 08:05:18.178761 25828 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.20.151.2:39107: connect: Connection refused (error 111)
W20240501 08:05:18.178978 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.20.151.2:39107: connect: Connection refused (error 111)
W20240501 08:05:18.182809 25828 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.20.151.3:45489: connect: Connection refused (error 111)
W20240501 08:05:18.183077 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.20.151.3:45489: connect: Connection refused (error 111)
W20240501 08:05:19.194098 25828 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.20.151.1:40455: connect: Connection refused (error 111)
W20240501 08:05:19.199752 25828 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.20.151.1:40455: connect: Connection refused (error 111)
W20240501 08:05:19.200042 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:40455: connect: Connection refused (error 111)
W20240501 08:05:19.204025 25828 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.20.151.2:39107: connect: Connection refused (error 111)
W20240501 08:05:19.204272 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.20.151.2:39107: connect: Connection refused (error 111)
W20240501 08:05:19.207933 25828 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.20.151.3:45489: connect: Connection refused (error 111)
W20240501 08:05:19.208175 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.20.151.3:45489: connect: Connection refused (error 111)
W20240501 08:05:20.210188 25828 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 412c21db44284526856da410d118cd9a not available for placement
W20240501 08:05:21.212070 25828 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 412c21db44284526856da410d118cd9a not available for placement
I20240501 08:05:21.477026 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:05:21.483189 26194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:21.484650 26195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:21.485253 26197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:05:21.489223 21084 server_base.cc:1034] running on GCE node
I20240501 08:05:21.491097 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:05:21.491289 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:05:21.491420 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550721491409 us; error 0 us; skew 500 ppm
I20240501 08:05:21.491899 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:05:21.500164 21084 webserver.cc:460] Webserver started at http://127.20.151.1:37475/ using document root <none> and password file <none>
I20240501 08:05:21.500643 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:05:21.500811 21084 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240501 08:05:21.504288 21084 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:05:21.507155 26202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:21.507876 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 08:05:21.508288 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-0-root
uuid: "a8bdf0eecb024a989a57ea4a16594503"
format_stamp: "Formatted at 2024-05-01 08:05:13 on dist-test-slave-pwkb"
I20240501 08:05:21.508620 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:05:21.573020 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:05:21.574180 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:05:21.575727 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:05:21.578987 26209 ts_tablet_manager.cc:522] Loading tablet metadata (0/4 complete)
I20240501 08:05:21.598707 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240501 08:05:21.598955 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.021s user 0.001s sys 0.000s
I20240501 08:05:21.599159 21084 ts_tablet_manager.cc:574] Registering tablets (0/4 complete)
I20240501 08:05:21.603260 26209 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503: Bootstrap starting.
I20240501 08:05:21.612565 21084 ts_tablet_manager.cc:590] Registered 4 tablets
I20240501 08:05:21.612794 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.014s user 0.011s sys 0.000s
I20240501 08:05:21.612828 26209 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:21.613574 26209 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503: Bootstrap complete.
I20240501 08:05:21.614114 26209 ts_tablet_manager.cc:1367] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20240501 08:05:21.615831 26209 raft_consensus.cc:348] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.616322 26209 raft_consensus.cc:729] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8bdf0eecb024a989a57ea4a16594503, State: Initialized, Role: FOLLOWER
I20240501 08:05:21.616866 26209 consensus_queue.cc:260] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.619228 26209 ts_tablet_manager.cc:1398] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240501 08:05:21.620048 26209 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503: Bootstrap starting.
I20240501 08:05:21.630698 26209 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:21.631357 26209 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503: Bootstrap complete.
I20240501 08:05:21.631975 26209 ts_tablet_manager.cc:1367] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 08:05:21.633925 26209 raft_consensus.cc:348] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.634680 26209 raft_consensus.cc:729] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8bdf0eecb024a989a57ea4a16594503, State: Initialized, Role: FOLLOWER
I20240501 08:05:21.635454 26209 consensus_queue.cc:260] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.637207 26209 ts_tablet_manager.cc:1398] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:05:21.637897 26209 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: Bootstrap starting.
I20240501 08:05:21.648828 26209 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:05:21.649995 26209 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: Bootstrap complete.
I20240501 08:05:21.650756 26209 ts_tablet_manager.cc:1367] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20240501 08:05:21.653270 26209 raft_consensus.cc:348] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.654060 26209 raft_consensus.cc:729] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8bdf0eecb024a989a57ea4a16594503, State: Initialized, Role: FOLLOWER
I20240501 08:05:21.654620 26209 consensus_queue.cc:260] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.656160 26209 ts_tablet_manager.cc:1398] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:05:21.656950 26209 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503: Bootstrap starting.
I20240501 08:05:21.667814 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:40455
I20240501 08:05:21.667924 26209 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:21.667959 26270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:40455 every 8 connection(s)
I20240501 08:05:21.668843 26209 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503: Bootstrap complete.
I20240501 08:05:21.669392 26209 ts_tablet_manager.cc:1367] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240501 08:05:21.671319 26209 raft_consensus.cc:348] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
W20240501 08:05:21.671901 26272 tablet.cc:2369] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:05:21.672194 26209 raft_consensus.cc:729] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8bdf0eecb024a989a57ea4a16594503, State: Initialized, Role: FOLLOWER
I20240501 08:05:21.672322 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:05:21.673254 26209 consensus_queue.cc:260] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.676167 26209 ts_tablet_manager.cc:1398] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20240501 08:05:21.682292 26275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:21.685506 26276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:21.690483 26278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:05:21.692148 21084 server_base.cc:1034] running on GCE node
I20240501 08:05:21.692534 26271 heartbeater.cc:347] Connected to a master server at 127.20.151.62:44219
I20240501 08:05:21.692975 26271 heartbeater.cc:464] Registering TS with master...
I20240501 08:05:21.693270 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:05:21.693539 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:05:21.693778 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550721693761 us; error 0 us; skew 500 ppm
I20240501 08:05:21.693912 26271 heartbeater.cc:511] Master 127.20.151.62:44219 requested a full tablet report, sending...
I20240501 08:05:21.694581 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:05:21.697520 21084 webserver.cc:460] Webserver started at http://127.20.151.2:32865/ using document root <none> and password file <none>
I20240501 08:05:21.698032 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:05:21.698204 21084 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240501 08:05:21.698097 25778 ts_manager.cc:194] Re-registered known tserver with Master: a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455)
I20240501 08:05:21.702031 21084 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:05:21.704967 25778 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50034
I20240501 08:05:21.705262 26283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:21.706099 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:05:21.706431 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-1-root
uuid: "412c21db44284526856da410d118cd9a"
format_stamp: "Formatted at 2024-05-01 08:05:13 on dist-test-slave-pwkb"
I20240501 08:05:21.706794 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:05:21.707829 26271 heartbeater.cc:503] Master 127.20.151.62:44219 was elected leader, sending a full tablet report...
I20240501 08:05:21.736213 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:05:21.737346 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:05:21.738698 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:05:21.759894 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240501 08:05:21.760223 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.019s user 0.001s sys 0.000s
I20240501 08:05:21.764226 26290 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a: Bootstrap starting.
I20240501 08:05:21.774461 26290 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:21.775121 26290 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a: Bootstrap complete.
I20240501 08:05:21.775295 21084 ts_tablet_manager.cc:590] Registered 4 tablets
I20240501 08:05:21.775548 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.015s user 0.014s sys 0.000s
I20240501 08:05:21.775636 26290 ts_tablet_manager.cc:1367] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240501 08:05:21.777722 26290 raft_consensus.cc:348] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.778230 26290 raft_consensus.cc:729] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412c21db44284526856da410d118cd9a, State: Initialized, Role: FOLLOWER
I20240501 08:05:21.778784 26290 consensus_queue.cc:260] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.780853 26290 ts_tablet_manager.cc:1398] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 08:05:21.781499 26290 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a: Bootstrap starting.
I20240501 08:05:21.792091 26290 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:21.792760 26290 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a: Bootstrap complete.
I20240501 08:05:21.793296 26290 ts_tablet_manager.cc:1367] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240501 08:05:21.795321 26290 raft_consensus.cc:348] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.795889 26290 raft_consensus.cc:729] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412c21db44284526856da410d118cd9a, State: Initialized, Role: FOLLOWER
I20240501 08:05:21.796533 26290 consensus_queue.cc:260] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.798554 26290 ts_tablet_manager.cc:1398] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20240501 08:05:21.799244 26290 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a: Bootstrap starting.
I20240501 08:05:21.811324 26290 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:21.812397 26290 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a: Bootstrap complete.
I20240501 08:05:21.813302 26290 ts_tablet_manager.cc:1367] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.002s
I20240501 08:05:21.816406 26290 raft_consensus.cc:348] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.817214 26290 raft_consensus.cc:729] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412c21db44284526856da410d118cd9a, State: Initialized, Role: FOLLOWER
I20240501 08:05:21.817998 26290 consensus_queue.cc:260] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.820501 26290 ts_tablet_manager.cc:1398] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240501 08:05:21.821507 26290 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: Bootstrap starting.
I20240501 08:05:21.829218 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:39107
I20240501 08:05:21.829289 26350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:39107 every 8 connection(s)
I20240501 08:05:21.836758 21084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:05:21.840066 26290 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:05:21.840888 26290 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: Bootstrap complete.
I20240501 08:05:21.841521 26290 ts_tablet_manager.cc:1367] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.000s
I20240501 08:05:21.844542 26290 raft_consensus.cc:348] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.845438 26290 raft_consensus.cc:729] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412c21db44284526856da410d118cd9a, State: Initialized, Role: FOLLOWER
I20240501 08:05:21.846184 26290 consensus_queue.cc:260] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.847230 26351 heartbeater.cc:347] Connected to a master server at 127.20.151.62:44219
I20240501 08:05:21.847620 26351 heartbeater.cc:464] Registering TS with master...
I20240501 08:05:21.848501 26351 heartbeater.cc:511] Master 127.20.151.62:44219 requested a full tablet report, sending...
I20240501 08:05:21.848544 26290 ts_tablet_manager.cc:1398] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20240501 08:05:21.850409 26356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:05:21.852260 26355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:05:21.854303 25778 ts_manager.cc:194] Re-registered known tserver with Master: 412c21db44284526856da410d118cd9a (127.20.151.2:39107)
I20240501 08:05:21.855581 21084 server_base.cc:1034] running on GCE node
W20240501 08:05:21.856593 26358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:05:21.857342 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:05:21.857550 21084 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:05:21.857733 21084 hybrid_clock.cc:648] HybridClock initialized: now 1714550721857715 us; error 0 us; skew 500 ppm
I20240501 08:05:21.858201 21084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:05:21.859098 25778 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50042
I20240501 08:05:21.860728 21084 webserver.cc:460] Webserver started at http://127.20.151.3:41153/ using document root <none> and password file <none>
I20240501 08:05:21.861182 21084 fs_manager.cc:359] Metadata directory not provided
I20240501 08:05:21.861340 21084 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240501 08:05:21.861603 26351 heartbeater.cc:503] Master 127.20.151.62:44219 was elected leader, sending a full tablet report...
I20240501 08:05:21.865206 21084 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:05:21.868659 26363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:05:21.869509 21084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 08:05:21.869915 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-2-root
uuid: "f1b6af9bbeab4df9a81e15a2965e6650"
format_stamp: "Formatted at 2024-05-01 08:05:13 on dist-test-slave-pwkb"
I20240501 08:05:21.870175 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskqlq16C/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714550573434241-21084-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:05:21.887938 21084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:05:21.888984 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:05:21.890355 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:05:21.912093 21084 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240501 08:05:21.912380 21084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.020s user 0.001s sys 0.000s
I20240501 08:05:21.916445 26370 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap starting.
I20240501 08:05:21.926851 21084 ts_tablet_manager.cc:590] Registered 4 tablets
I20240501 08:05:21.927098 21084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.014s user 0.014s sys 0.000s
I20240501 08:05:21.928998 26370 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:05:21.929759 26370 tablet_bootstrap.cc:492] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap complete.
I20240501 08:05:21.930248 26370 ts_tablet_manager.cc:1367] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20240501 08:05:21.931989 26370 raft_consensus.cc:348] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.932667 26370 raft_consensus.cc:729] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b6af9bbeab4df9a81e15a2965e6650, State: Initialized, Role: FOLLOWER
I20240501 08:05:21.933473 26370 consensus_queue.cc:260] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.936071 26370 ts_tablet_manager.cc:1398] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 08:05:21.936698 26370 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap starting.
I20240501 08:05:21.949977 26370 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:05:21.951189 26370 tablet_bootstrap.cc:492] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap complete.
I20240501 08:05:21.951926 26370 ts_tablet_manager.cc:1367] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20240501 08:05:21.954730 26370 raft_consensus.cc:348] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.955690 26370 raft_consensus.cc:729] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b6af9bbeab4df9a81e15a2965e6650, State: Initialized, Role: FOLLOWER
I20240501 08:05:21.956277 26370 consensus_queue.cc:260] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.957824 26370 ts_tablet_manager.cc:1398] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 08:05:21.958554 26370 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap starting.
I20240501 08:05:21.971900 26370 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:21.972904 26370 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap complete.
I20240501 08:05:21.973778 26370 ts_tablet_manager.cc:1367] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650: Time spent bootstrapping tablet: real 0.015s user 0.015s sys 0.000s
I20240501 08:05:21.976646 26370 raft_consensus.cc:348] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.977555 26370 raft_consensus.cc:729] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b6af9bbeab4df9a81e15a2965e6650, State: Initialized, Role: FOLLOWER
I20240501 08:05:21.978430 26370 consensus_queue.cc:260] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.978950 21084 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:45489
I20240501 08:05:21.979032 26430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:45489 every 8 connection(s)
I20240501 08:05:21.980649 26370 ts_tablet_manager.cc:1398] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:05:21.981423 26370 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap starting.
I20240501 08:05:21.994153 26370 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:21.994967 26370 tablet_bootstrap.cc:492] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650: Bootstrap complete.
I20240501 08:05:21.995074 26431 heartbeater.cc:347] Connected to a master server at 127.20.151.62:44219
I20240501 08:05:21.995702 26431 heartbeater.cc:464] Registering TS with master...
I20240501 08:05:21.995769 26370 ts_tablet_manager.cc:1367] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240501 08:05:21.996629 26431 heartbeater.cc:511] Master 127.20.151.62:44219 requested a full tablet report, sending...
I20240501 08:05:21.998121 26370 raft_consensus.cc:348] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:21.998960 26370 raft_consensus.cc:729] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b6af9bbeab4df9a81e15a2965e6650, State: Initialized, Role: FOLLOWER
I20240501 08:05:21.999783 26370 consensus_queue.cc:260] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:22.000337 25778 ts_manager.cc:194] Re-registered known tserver with Master: f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:22.002290 26370 ts_tablet_manager.cc:1398] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 08:05:22.005584 25778 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50046
I20240501 08:05:22.008147 26431 heartbeater.cc:503] Master 127.20.151.62:44219 was elected leader, sending a full tablet report...
W20240501 08:05:22.068487 26174 debug-util.cc:398] Leaking SignalData structure 0x7b08000b9280 after lost signal to thread 21087
W20240501 08:05:22.070159 26174 debug-util.cc:398] Leaking SignalData structure 0x7b08001caf40 after lost signal to thread 26177
W20240501 08:05:22.233004 25828 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica f1b6af9bbeab4df9a81e15a2965e6650 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } } }
W20240501 08:05:22.237576 25828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c22bf70d91434f688f21d7a288a2d355, TS 412c21db44284526856da410d118cd9a -> TS 594bfa72576e4461aa048eb7474768fa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
W20240501 08:05:22.238219 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c22bf70d91434f688f21d7a288a2d355, TS 412c21db44284526856da410d118cd9a -> TS 594bfa72576e4461aa048eb7474768fa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
W20240501 08:05:22.250299 25828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b049db705b3242ecbd1b6483bfee9813, TS f1b6af9bbeab4df9a81e15a2965e6650 -> TS 594bfa72576e4461aa048eb7474768fa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
W20240501 08:05:22.250567 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b049db705b3242ecbd1b6483bfee9813, TS f1b6af9bbeab4df9a81e15a2965e6650 -> TS 594bfa72576e4461aa048eb7474768fa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
W20240501 08:05:22.253489 25828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1b9e3f16ca2242069311a4aa01e1a9d3, TS a8bdf0eecb024a989a57ea4a16594503 -> TS 594bfa72576e4461aa048eb7474768fa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
W20240501 08:05:22.253747 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 1b9e3f16ca2242069311a4aa01e1a9d3, TS a8bdf0eecb024a989a57ea4a16594503 -> TS 594bfa72576e4461aa048eb7474768fa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:22.797614 26438 raft_consensus.cc:482] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:22.798117 26438 raft_consensus.cc:504] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:22.800102 26438 leader_election.cc:288] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 412c21db44284526856da410d118cd9a (127.20.151.2:39107), f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:22.814798 26405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22bf70d91434f688f21d7a288a2d355" candidate_uuid: "a8bdf0eecb024a989a57ea4a16594503" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" is_pre_election: true
I20240501 08:05:22.815559 26405 raft_consensus.cc:2396] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a8bdf0eecb024a989a57ea4a16594503 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 08:05:22.817175 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22bf70d91434f688f21d7a288a2d355" candidate_uuid: "a8bdf0eecb024a989a57ea4a16594503" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412c21db44284526856da410d118cd9a" is_pre_election: true
I20240501 08:05:22.817708 26326 raft_consensus.cc:2454] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8bdf0eecb024a989a57ea4a16594503 in term 1.
I20240501 08:05:22.818590 26205 leader_election.cc:302] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 412c21db44284526856da410d118cd9a, a8bdf0eecb024a989a57ea4a16594503; no voters: f1b6af9bbeab4df9a81e15a2965e6650
I20240501 08:05:22.819265 26438 raft_consensus.cc:2781] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:05:22.819602 26438 raft_consensus.cc:482] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:22.819862 26438 raft_consensus.cc:3037] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:22.824366 26438 raft_consensus.cc:504] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:22.825930 26438 leader_election.cc:288] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [CANDIDATE]: Term 2 election: Requested vote from peers 412c21db44284526856da410d118cd9a (127.20.151.2:39107), f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:22.826640 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22bf70d91434f688f21d7a288a2d355" candidate_uuid: "a8bdf0eecb024a989a57ea4a16594503" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412c21db44284526856da410d118cd9a"
I20240501 08:05:22.827054 26405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22bf70d91434f688f21d7a288a2d355" candidate_uuid: "a8bdf0eecb024a989a57ea4a16594503" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1b6af9bbeab4df9a81e15a2965e6650"
I20240501 08:05:22.827250 26326 raft_consensus.cc:3037] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:22.827634 26405 raft_consensus.cc:3037] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:22.834039 26326 raft_consensus.cc:2454] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8bdf0eecb024a989a57ea4a16594503 in term 2.
I20240501 08:05:22.834368 26405 raft_consensus.cc:2396] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a8bdf0eecb024a989a57ea4a16594503 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 08:05:22.835098 26205 leader_election.cc:302] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c21db44284526856da410d118cd9a, a8bdf0eecb024a989a57ea4a16594503; no voters:
I20240501 08:05:22.835822 26438 raft_consensus.cc:2781] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:05:22.837066 26438 raft_consensus.cc:686] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 2 LEADER]: Becoming Leader. State: Replica: a8bdf0eecb024a989a57ea4a16594503, State: Running, Role: LEADER
I20240501 08:05:22.837975 26438 consensus_queue.cc:237] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:22.847276 25778 catalog_manager.cc:5484] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 reported cstate change: term changed from 1 to 2, leader changed from f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3) to a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1). New cstate: current_term: 2 leader_uuid: "a8bdf0eecb024a989a57ea4a16594503" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:22.866101 26438 raft_consensus.cc:482] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:22.866567 26438 raft_consensus.cc:504] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:22.868214 26438 leader_election.cc:288] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 412c21db44284526856da410d118cd9a (127.20.151.2:39107), f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:22.868995 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aa10ca2f384362841703c074e8ea3b" candidate_uuid: "a8bdf0eecb024a989a57ea4a16594503" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "412c21db44284526856da410d118cd9a" is_pre_election: true
I20240501 08:05:22.869589 26326 raft_consensus.cc:2454] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8bdf0eecb024a989a57ea4a16594503 in term 1.
I20240501 08:05:22.869413 26405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aa10ca2f384362841703c074e8ea3b" candidate_uuid: "a8bdf0eecb024a989a57ea4a16594503" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" is_pre_election: true
I20240501 08:05:22.870220 26405 raft_consensus.cc:2454] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8bdf0eecb024a989a57ea4a16594503 in term 1.
I20240501 08:05:22.870476 26205 leader_election.cc:302] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c21db44284526856da410d118cd9a, a8bdf0eecb024a989a57ea4a16594503; no voters:
I20240501 08:05:22.871366 26438 raft_consensus.cc:2781] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:05:22.871723 26438 raft_consensus.cc:482] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:22.872035 26438 raft_consensus.cc:3037] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:22.877367 26438 raft_consensus.cc:504] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:22.878944 26438 leader_election.cc:288] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [CANDIDATE]: Term 2 election: Requested vote from peers 412c21db44284526856da410d118cd9a (127.20.151.2:39107), f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:22.879685 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aa10ca2f384362841703c074e8ea3b" candidate_uuid: "a8bdf0eecb024a989a57ea4a16594503" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "412c21db44284526856da410d118cd9a"
I20240501 08:05:22.880167 26326 raft_consensus.cc:3037] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:22.880273 26405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aa10ca2f384362841703c074e8ea3b" candidate_uuid: "a8bdf0eecb024a989a57ea4a16594503" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f1b6af9bbeab4df9a81e15a2965e6650"
I20240501 08:05:22.880858 26405 raft_consensus.cc:3037] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:22.886720 26326 raft_consensus.cc:2454] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8bdf0eecb024a989a57ea4a16594503 in term 2.
I20240501 08:05:22.887090 26405 raft_consensus.cc:2454] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8bdf0eecb024a989a57ea4a16594503 in term 2.
I20240501 08:05:22.887902 26205 leader_election.cc:302] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c21db44284526856da410d118cd9a, a8bdf0eecb024a989a57ea4a16594503; no voters:
I20240501 08:05:22.888530 26438 raft_consensus.cc:2781] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:05:22.888892 26438 raft_consensus.cc:686] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 2 LEADER]: Becoming Leader. State: Replica: a8bdf0eecb024a989a57ea4a16594503, State: Running, Role: LEADER
I20240501 08:05:22.889609 26438 consensus_queue.cc:237] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:22.895726 25778 catalog_manager.cc:5484] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "a8bdf0eecb024a989a57ea4a16594503" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:22.934732 26450 raft_consensus.cc:482] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:22.935173 26450 raft_consensus.cc:504] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:22.937016 26450 leader_election.cc:288] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455), f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:22.949004 26405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9e3f16ca2242069311a4aa01e1a9d3" candidate_uuid: "412c21db44284526856da410d118cd9a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" is_pre_election: true
I20240501 08:05:22.950016 26405 raft_consensus.cc:2396] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 412c21db44284526856da410d118cd9a for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 08:05:22.951555 26244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9e3f16ca2242069311a4aa01e1a9d3" candidate_uuid: "412c21db44284526856da410d118cd9a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8bdf0eecb024a989a57ea4a16594503" is_pre_election: true
I20240501 08:05:22.952272 26244 raft_consensus.cc:2454] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412c21db44284526856da410d118cd9a in term 1.
I20240501 08:05:22.953261 26284 leader_election.cc:302] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 412c21db44284526856da410d118cd9a, a8bdf0eecb024a989a57ea4a16594503; no voters: f1b6af9bbeab4df9a81e15a2965e6650
I20240501 08:05:22.953933 26450 raft_consensus.cc:2781] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:05:22.954272 26450 raft_consensus.cc:482] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:22.954501 26450 raft_consensus.cc:3037] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:22.958916 26450 raft_consensus.cc:504] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:22.960384 26450 leader_election.cc:288] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [CANDIDATE]: Term 2 election: Requested vote from peers a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455), f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:22.961392 26405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9e3f16ca2242069311a4aa01e1a9d3" candidate_uuid: "412c21db44284526856da410d118cd9a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1b6af9bbeab4df9a81e15a2965e6650"
I20240501 08:05:22.961243 26244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9e3f16ca2242069311a4aa01e1a9d3" candidate_uuid: "412c21db44284526856da410d118cd9a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8bdf0eecb024a989a57ea4a16594503"
I20240501 08:05:22.961926 26405 raft_consensus.cc:3037] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:22.962061 26244 raft_consensus.cc:3037] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:22.966356 26405 raft_consensus.cc:2396] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 412c21db44284526856da410d118cd9a for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 08:05:22.969640 26244 raft_consensus.cc:2454] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412c21db44284526856da410d118cd9a in term 2.
I20240501 08:05:22.970690 26284 leader_election.cc:302] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 412c21db44284526856da410d118cd9a, a8bdf0eecb024a989a57ea4a16594503; no voters: f1b6af9bbeab4df9a81e15a2965e6650
I20240501 08:05:22.971341 26450 raft_consensus.cc:2781] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:05:22.972584 26450 raft_consensus.cc:686] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 2 LEADER]: Becoming Leader. State: Replica: 412c21db44284526856da410d118cd9a, State: Running, Role: LEADER
I20240501 08:05:22.973487 26450 consensus_queue.cc:237] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:22.981839 25778 catalog_manager.cc:5484] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a reported cstate change: term changed from 1 to 2, leader changed from f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3) to 412c21db44284526856da410d118cd9a (127.20.151.2). New cstate: current_term: 2 leader_uuid: "412c21db44284526856da410d118cd9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } health_report { overall_health: UNKNOWN } } }
W20240501 08:05:23.271338 25828 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240501 08:05:23.275234 25828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d8aa10ca2f384362841703c074e8ea3b, TS 412c21db44284526856da410d118cd9a -> TS 594bfa72576e4461aa048eb7474768fa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
W20240501 08:05:23.275475 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d8aa10ca2f384362841703c074e8ea3b, TS 412c21db44284526856da410d118cd9a -> TS 594bfa72576e4461aa048eb7474768fa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
W20240501 08:05:23.278599 25828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c22bf70d91434f688f21d7a288a2d355, TS f1b6af9bbeab4df9a81e15a2965e6650 -> TS 594bfa72576e4461aa048eb7474768fa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
W20240501 08:05:23.278834 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c22bf70d91434f688f21d7a288a2d355, TS f1b6af9bbeab4df9a81e15a2965e6650 -> TS 594bfa72576e4461aa048eb7474768fa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
W20240501 08:05:23.281852 25828 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b049db705b3242ecbd1b6483bfee9813, TS a8bdf0eecb024a989a57ea4a16594503 -> TS 594bfa72576e4461aa048eb7474768fa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
W20240501 08:05:23.282073 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b049db705b3242ecbd1b6483bfee9813, TS a8bdf0eecb024a989a57ea4a16594503 -> TS 594bfa72576e4461aa048eb7474768fa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:23.301012 26405 raft_consensus.cc:1261] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Refusing update from remote peer a8bdf0eecb024a989a57ea4a16594503: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240501 08:05:23.301375 26405 pending_rounds.cc:76] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Aborting all ops after (but not including) 0
I20240501 08:05:23.301640 26405 pending_rounds.cc:97] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240501 08:05:23.301893 26405 raft_consensus.cc:2866] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:05:23.303210 26438 consensus_queue.cc:1035] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:05:23.310283 26326 raft_consensus.cc:1261] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Refusing update from remote peer a8bdf0eecb024a989a57ea4a16594503: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 08:05:23.311959 26438 consensus_queue.cc:1035] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:05:23.363775 26326 raft_consensus.cc:1261] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Refusing update from remote peer a8bdf0eecb024a989a57ea4a16594503: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:05:23.365175 26438 consensus_queue.cc:1035] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:05:23.385571 26405 raft_consensus.cc:1261] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Refusing update from remote peer a8bdf0eecb024a989a57ea4a16594503: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:05:23.387904 26465 consensus_queue.cc:1035] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:05:23.431985 26454 raft_consensus.cc:482] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:05:23.432616 26454 raft_consensus.cc:504] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:23.434916 26454 leader_election.cc:288] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455), f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:23.435640 26244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b049db705b3242ecbd1b6483bfee9813" candidate_uuid: "412c21db44284526856da410d118cd9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a8bdf0eecb024a989a57ea4a16594503" is_pre_election: true
I20240501 08:05:23.435926 26405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b049db705b3242ecbd1b6483bfee9813" candidate_uuid: "412c21db44284526856da410d118cd9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" is_pre_election: true
I20240501 08:05:23.436177 26244 raft_consensus.cc:2454] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412c21db44284526856da410d118cd9a in term 1.
I20240501 08:05:23.436652 26405 raft_consensus.cc:2454] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412c21db44284526856da410d118cd9a in term 1.
I20240501 08:05:23.437136 26284 leader_election.cc:302] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c21db44284526856da410d118cd9a, a8bdf0eecb024a989a57ea4a16594503; no voters:
I20240501 08:05:23.437873 26454 raft_consensus.cc:2781] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:05:23.438234 26454 raft_consensus.cc:482] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:05:23.438576 26454 raft_consensus.cc:3037] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:23.445861 26454 raft_consensus.cc:504] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:23.447839 26454 leader_election.cc:288] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [CANDIDATE]: Term 2 election: Requested vote from peers a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455), f1b6af9bbeab4df9a81e15a2965e6650 (127.20.151.3:45489)
I20240501 08:05:23.449095 26244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b049db705b3242ecbd1b6483bfee9813" candidate_uuid: "412c21db44284526856da410d118cd9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a8bdf0eecb024a989a57ea4a16594503"
I20240501 08:05:23.449784 26244 raft_consensus.cc:3037] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:23.449770 26405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b049db705b3242ecbd1b6483bfee9813" candidate_uuid: "412c21db44284526856da410d118cd9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f1b6af9bbeab4df9a81e15a2965e6650"
I20240501 08:05:23.450582 26405 raft_consensus.cc:3037] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:05:23.454437 26244 raft_consensus.cc:2454] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412c21db44284526856da410d118cd9a in term 2.
I20240501 08:05:23.455282 26284 leader_election.cc:302] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c21db44284526856da410d118cd9a, a8bdf0eecb024a989a57ea4a16594503; no voters:
I20240501 08:05:23.455863 26454 raft_consensus.cc:2781] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:05:23.456208 26454 raft_consensus.cc:686] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 2 LEADER]: Becoming Leader. State: Replica: 412c21db44284526856da410d118cd9a, State: Running, Role: LEADER
I20240501 08:05:23.457226 26454 consensus_queue.cc:237] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } }
I20240501 08:05:23.457526 26405 raft_consensus.cc:2454] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412c21db44284526856da410d118cd9a in term 2.
I20240501 08:05:23.464766 25777 catalog_manager.cc:5484] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "412c21db44284526856da410d118cd9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20240501 08:05:23.466399 26405 raft_consensus.cc:1261] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Refusing update from remote peer 412c21db44284526856da410d118cd9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240501 08:05:23.466698 26405 pending_rounds.cc:76] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Aborting all ops after (but not including) 0
I20240501 08:05:23.466928 26405 pending_rounds.cc:97] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240501 08:05:23.467119 26405 raft_consensus.cc:2866] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:05:23.468240 26454 consensus_queue.cc:1035] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:05:23.479586 26244 raft_consensus.cc:1261] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Refusing update from remote peer 412c21db44284526856da410d118cd9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 08:05:23.481093 26450 consensus_queue.cc:1035] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:05:23.934808 26405 raft_consensus.cc:1261] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Refusing update from remote peer 412c21db44284526856da410d118cd9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:05:23.936084 26450 consensus_queue.cc:1035] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:05:23.946547 26244 raft_consensus.cc:1261] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Refusing update from remote peer 412c21db44284526856da410d118cd9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:05:23.948127 26454 consensus_queue.cc:1035] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:05:24.317023 26326 consensus_queue.cc:237] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } attrs { replace: true } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } }
I20240501 08:05:24.333470 26404 raft_consensus.cc:1261] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Refusing update from remote peer 412c21db44284526856da410d118cd9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:05:24.335728 26244 raft_consensus.cc:1261] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Refusing update from remote peer 412c21db44284526856da410d118cd9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:05:24.337323 26469 consensus_queue.cc:1035] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 08:05:24.338274 26454 consensus_queue.cc:1035] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 08:05:24.346042 26454 raft_consensus.cc:2932] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 594bfa72576e4461aa048eb7474768fa (127.20.151.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } attrs { replace: true } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } } }
W20240501 08:05:24.354434 26286 consensus_peers.cc:485] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a -> Peer 594bfa72576e4461aa048eb7474768fa (127.20.151.4:40155): Couldn't send request to peer 594bfa72576e4461aa048eb7474768fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b049db705b3242ecbd1b6483bfee9813. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:24.356230 26405 raft_consensus.cc:2932] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 594bfa72576e4461aa048eb7474768fa (127.20.151.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } attrs { replace: true } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } } }
I20240501 08:05:24.359333 26244 raft_consensus.cc:2932] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 594bfa72576e4461aa048eb7474768fa (127.20.151.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } attrs { replace: true } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } } }
I20240501 08:05:24.366811 26243 consensus_queue.cc:237] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } attrs { replace: true } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } }
I20240501 08:05:24.380478 26405 raft_consensus.cc:1261] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Refusing update from remote peer a8bdf0eecb024a989a57ea4a16594503: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:05:24.382155 26438 consensus_queue.cc:1035] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 08:05:24.383884 25778 catalog_manager.cc:5484] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 reported cstate change: config changed from index -1 to 3, NON_VOTER 594bfa72576e4461aa048eb7474768fa (127.20.151.4) added. New cstate: current_term: 2 leader_uuid: "412c21db44284526856da410d118cd9a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } attrs { replace: true } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } } }
I20240501 08:05:24.393748 26465 raft_consensus.cc:2932] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 594bfa72576e4461aa048eb7474768fa (127.20.151.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } attrs { replace: true } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } } }
I20240501 08:05:24.395866 26404 raft_consensus.cc:2932] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 594bfa72576e4461aa048eb7474768fa (127.20.151.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } attrs { replace: true } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } } }
I20240501 08:05:24.413038 26326 raft_consensus.cc:1261] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Refusing update from remote peer a8bdf0eecb024a989a57ea4a16594503: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:05:24.414976 26476 consensus_queue.cc:1035] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240501 08:05:24.430696 26205 consensus_peers.cc:485] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 -> Peer 594bfa72576e4461aa048eb7474768fa (127.20.151.4:40155): Couldn't send request to peer 594bfa72576e4461aa048eb7474768fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8aa10ca2f384362841703c074e8ea3b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:24.433547 26324 consensus_queue.cc:237] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } }
I20240501 08:05:24.440812 26326 raft_consensus.cc:2932] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 594bfa72576e4461aa048eb7474768fa (127.20.151.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } attrs { replace: true } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } } }
W20240501 08:05:24.443019 26286 consensus_peers.cc:485] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a -> Peer 594bfa72576e4461aa048eb7474768fa (127.20.151.4:40155): Couldn't send request to peer 594bfa72576e4461aa048eb7474768fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b9e3f16ca2242069311a4aa01e1a9d3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:24.445807 26404 raft_consensus.cc:1261] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Refusing update from remote peer 412c21db44284526856da410d118cd9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:05:24.447947 26450 consensus_queue.cc:1035] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 08:05:24.460170 26454 raft_consensus.cc:2932] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 594bfa72576e4461aa048eb7474768fa (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } } }
I20240501 08:05:24.468403 26243 raft_consensus.cc:1261] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Refusing update from remote peer 412c21db44284526856da410d118cd9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:05:24.470343 26450 consensus_queue.cc:1035] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:05:24.465724 26404 raft_consensus.cc:2932] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 594bfa72576e4461aa048eb7474768fa (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } } }
I20240501 08:05:24.484534 25778 catalog_manager.cc:5484] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a reported cstate change: config changed from index -1 to 3, NON_VOTER 594bfa72576e4461aa048eb7474768fa (127.20.151.4) added. New cstate: current_term: 2 leader_uuid: "a8bdf0eecb024a989a57ea4a16594503" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } attrs { replace: true } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } } }
I20240501 08:05:24.493650 26243 raft_consensus.cc:2932] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 594bfa72576e4461aa048eb7474768fa (127.20.151.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } } }
I20240501 08:05:24.505514 25778 catalog_manager.cc:5484] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 reported cstate change: config changed from index -1 to 2, NON_VOTER 594bfa72576e4461aa048eb7474768fa (127.20.151.4) added. New cstate: current_term: 2 leader_uuid: "412c21db44284526856da410d118cd9a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a8bdf0eecb024a989a57ea4a16594503" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40455 } } peers { permanent_uuid: "412c21db44284526856da410d118cd9a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39107 } } peers { permanent_uuid: "f1b6af9bbeab4df9a81e15a2965e6650" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "594bfa72576e4461aa048eb7474768fa" member_type: NON_VOTER last_known_addr { host: "127.20.151.4" port: 40155 } attrs { promote: true } } }
I20240501 08:05:24.795509 26483 ts_tablet_manager.cc:902] T d8aa10ca2f384362841703c074e8ea3b P 594bfa72576e4461aa048eb7474768fa: Initiating tablet copy from peer a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455)
I20240501 08:05:24.805773 26483 tablet_copy_client.cc:308] T d8aa10ca2f384362841703c074e8ea3b P 594bfa72576e4461aa048eb7474768fa: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.1:40455
I20240501 08:05:24.836915 26490 ts_tablet_manager.cc:902] T b049db705b3242ecbd1b6483bfee9813 P 594bfa72576e4461aa048eb7474768fa: Initiating tablet copy from peer 412c21db44284526856da410d118cd9a (127.20.151.2:39107)
I20240501 08:05:24.836973 26255 tablet_copy_service.cc:140] P a8bdf0eecb024a989a57ea4a16594503: Received BeginTabletCopySession request for tablet d8aa10ca2f384362841703c074e8ea3b from peer 594bfa72576e4461aa048eb7474768fa ({username='slave'} at 127.0.0.1:42960)
I20240501 08:05:24.838865 26255 tablet_copy_service.cc:161] P a8bdf0eecb024a989a57ea4a16594503: Beginning new tablet copy session on tablet d8aa10ca2f384362841703c074e8ea3b from peer 594bfa72576e4461aa048eb7474768fa at {username='slave'} at 127.0.0.1:42960: session id = 594bfa72576e4461aa048eb7474768fa-d8aa10ca2f384362841703c074e8ea3b
F20240501 08:05:24.838536 26490 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-177f26bc23d44e0aad642cf1be6b376e.queue_time_us, tablet-download-pool-177f26bc23d44e0aad642cf1be6b376e.queue_time_us) (tablet-download-pool-177f26bc23d44e0aad642cf1be6b376e.run_cpu_time_us, tablet-download-pool-177f26bc23d44e0aad642cf1be6b376e.run_cpu_time_us) (tablet-download-pool-177f26bc23d44e0aad642cf1be6b376e.run_wall_time_us, tablet-download-pool-177f26bc23d44e0aad642cf1be6b376e.run_wall_time_us) (tablet-download-pool-19ccc66f74df4b3ebc1fceb2499c0308.queue_time_us, tablet-download-pool-19ccc66f74df4b3ebc1fceb2499c0308.queue_time_us) (tablet-download-pool-19ccc66f74df4b3ebc1fceb2499c0308.run_cpu_time_us, tablet-download-pool-19ccc66f74df4b3ebc1fceb2499c0308.run_cpu_time_us) (tablet-download-pool-19ccc66f74df4b3ebc1fceb2499c0308.run_wall_time_us, tablet-download-pool-19ccc66f74df4b3ebc1fceb2499c0308.run_wall_time_us) (tablet-download-pool-26841e9dcbca4d6d995fba9546ada63c.queue_time_us, tablet-download-pool-26841e9dcbca4d6d995fba9546ada63c.queue_time_us) (tablet-download-pool-26841e9dcbca4d6d995fba9546ada63c.run_cpu_time_us, tablet-download-pool-26841e9dcbca4d6d995fba9546ada63c.run_cpu_time_us) (tablet-download-pool-26841e9dcbca4d6d995fba9546ada63c.run_wall_time_us, tablet-download-pool-26841e9dcbca4d6d995fba9546ada63c.run_wall_time_us) (tablet-download-pool-26dc8e957a2747528356007dab573637.queue_time_us, tablet-download-pool-26dc8e957a2747528356007dab573637.queue_time_us) (tablet-download-pool-26dc8e957a2747528356007dab573637.run_cpu_time_us, tablet-download-pool-26dc8e957a2747528356007dab573637.run_cpu_time_us) (tablet-download-pool-26dc8e957a2747528356007dab573637.run_wall_time_us, tablet-download-pool-26dc8e957a2747528356007dab573637.run_wall_time_us) (tablet-download-pool-44bf2a078f494657bceaca86c03dbbfc.queue_time_us, tablet-download-pool-44bf2a078f494657bceaca86c03dbbfc.queue_time_us) (tablet-download-pool-44bf2a078f494657bceaca86c03dbbfc.run_cpu_time_us, tablet-download-pool-44bf2a078f494657bceaca86c03dbbfc.run_cpu_time_us) (tablet-download-pool-44bf2a078f494657bceaca86c03dbbfc.run_wall_time_us, tablet-download-pool-44bf2a078f494657bceaca86c03dbbfc.run_wall_time_us) (tablet-download-pool-63c05c8273a34209a6078c89d59bd01d.queue_time_us, tablet-download-pool-63c05c8273a34209a6078c89d59bd01d.queue_time_us) (tablet-download-pool-63c05c8273a34209a6078c89d59bd01d.run_cpu_time_us, tablet-download-pool-63c05c8273a34209a6078c89d59bd01d.run_cpu_time_us) (tablet-download-pool-63c05c8273a34209a6078c89d59bd01d.run_wall_time_us, tablet-download-pool-63c05c8273a34209a6078c89d59bd01d.run_wall_time_us) (tablet-download-pool-9dc237bb85424c3d968dd18ada46214c.queue_time_us, tablet-download-pool-9dc237bb85424c3d968dd18ada46214c.queue_time_us) (tablet-download-pool-9dc237bb85424c3d968dd18ada46214c.run_cpu_time_us, tablet-download-pool-9dc237bb85424c3d968dd18ada46214c.run_cpu_time_us) (tablet-download-pool-9dc237bb85424c3d968dd18ada46214c.run_wall_time_us, tablet-download-pool-9dc237bb85424c3d968dd18ada46214c.run_wall_time_us) (tablet-download-pool-aefa79f68d754a96b24ca2e3dbe1c4fe.queue_time_us, tablet-download-pool-aefa79f68d754a96b24ca2e3dbe1c4fe.queue_time_us) (tablet-download-pool-aefa79f68d754a96b24ca2e3dbe1c4fe.run_cpu_time_us, tablet-download-pool-aefa79f68d754a96b24ca2e3dbe1c4fe.run_cpu_time_us) (tablet-download-pool-aefa79f68d754a96b24ca2e3dbe1c4fe.run_wall_time_us, tablet-download-pool-aefa79f68d754a96b24ca2e3dbe1c4fe.run_wall_time_us) (tablet-download-pool-b049db705b3242ecbd1b6483bfee9813.queue_time_us, tablet-download-pool-b049db705b3242ecbd1b6483bfee9813.queue_time_us) (tablet-download-pool-b5710bd5ed314c1c8794e04c25b48b3e.queue_time_us, tablet-download-pool-b5710bd5ed314c1c8794e04c25b48b3e.queue_time_us) (tablet-download-pool-b5710bd5ed314c1c8794e04c25b48b3e.run_cpu_time_us, tablet-download-pool-b5710bd5ed314c1c8794e04c25b48b3e.run_cpu_time_us) (tablet-download-pool-b5710bd5ed314c1c8794e04c25b48b3e.run_wall_time_us, tablet-download-pool-b5710bd5ed314c1c8794e04c25b48b3e.run_wall_time_us) (tablet-download-pool-d59079b3a4f44463b337d12cf267ca54.queue_time_us, tablet-download-pool-d59079b3a4f44463b337d12cf267ca54.queue_time_us) (tablet-download-pool-d59079b3a4f44463b337d12cf267ca54.run_cpu_time_us, tablet-download-pool-d59079b3a4f44463b337d12cf267ca54.run_cpu_time_us) (tablet-download-pool-d59079b3a4f44463b337d12cf267ca54.run_wall_time_us, tablet-download-pool-d59079b3a4f44463b337d12cf267ca54.run_wall_time_us) (tablet-download-pool-d8aa10ca2f384362841703c074e8ea3b.queue_time_us, tablet-download-pool-d8aa10ca2f384362841703c074e8ea3b.queue_time_us) (tablet-download-pool-d8aa10ca2f384362841703c074e8ea3b.run_cpu_time_us, tablet-download-pool-d8aa10ca2f384362841703c074e8ea3b.run_cpu_time_us) (tablet-download-pool-d8aa10ca2f384362841703c074e8ea3b.run_wall_time_us, tablet-download-pool-d8aa10ca2f384362841703c074e8ea3b.run_wall_time_us) (tablet-download-pool-f652910a32a348339c345ce75ce79be7.queue_time_us, tablet-download-pool-f652910a32a348339c345ce75ce79be7.queue_time_us) (tablet-download-pool-f652910a32a348339c345ce75ce79be7.run_cpu_time_us, tablet-download-pool-f652910a32a348339c345ce75ce79be7.run_cpu_time_us) (tablet-download-pool-f652910a32a348339c345ce75ce79be7.run_wall_time_us, tablet-download-pool-f652910a32a348339c345ce75ce79be7.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240501 08:05:24.846130 26491 ts_tablet_manager.cc:902] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 594bfa72576e4461aa048eb7474768fa: Initiating tablet copy from peer 412c21db44284526856da410d118cd9a (127.20.151.2:39107)
*** Check failure stack trace: ***
*** Aborted at 1714550724 (unix time) try "date -d @1714550724" if you are using GNU date ***
I20240501 08:05:24.851042 26255 tablet_copy_source_session.cc:205] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:05:24.860957 26483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8aa10ca2f384362841703c074e8ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
PC: @ 0x0 (unknown)
I20240501 08:05:24.884784 26483 tablet_copy_client.cc:783] T d8aa10ca2f384362841703c074e8ea3b P 594bfa72576e4461aa048eb7474768fa: tablet copy: Starting download of 0 data blocks...
I20240501 08:05:24.886035 26483 tablet_copy_client.cc:647] T d8aa10ca2f384362841703c074e8ea3b P 594bfa72576e4461aa048eb7474768fa: tablet copy: Starting download of 1 WAL segments...
*** SIGABRT (@0x3e80000525c) received by PID 21084 (TID 0x7f214aaf7700) from PID 21084; stack trace: ***
@ 0x2e41a0 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e6683 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20240501 08:05:24.890741 26483 tablet_copy_client.cc:523] T d8aa10ca2f384362841703c074e8ea3b P 594bfa72576e4461aa048eb7474768fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:24.901317 26483 tablet_bootstrap.cc:492] T d8aa10ca2f384362841703c074e8ea3b P 594bfa72576e4461aa048eb7474768fa: Bootstrap starting.
@ 0x7f221bca7980 (unknown) at ??:0
@ 0x7f221521bfb7 gsignal at ??:0
@ 0x7f221521d921 abort at ??:0
@ 0x2ea196 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f2218011da4 google::LogMessage::SendToLog() at ??:0
@ 0x7f2218012950 google::LogMessage::Flush() at ??:0
@ 0x7f2218017a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f221899f3c7 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f221897594f kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f22189730a5 kudu::ThreadPoolBuilder::Build() at ??:0
W20240501 08:05:25.173449 25811 debug-util.cc:398] Leaking SignalData structure 0x7b0800120260 after lost signal to thread 21087
W20240501 08:05:25.174863 25811 debug-util.cc:398] Leaking SignalData structure 0x7b0800043400 after lost signal to thread 25814
W20240501 08:05:25.176285 25811 debug-util.cc:398] Leaking SignalData structure 0x7b0800147140 after lost signal to thread 26177
W20240501 08:05:25.178347 25811 debug-util.cc:398] Leaking SignalData structure 0x7b08000a3500 after lost signal to thread 26350
W20240501 08:05:25.179553 25811 debug-util.cc:398] Leaking SignalData structure 0x7b08001ad740 after lost signal to thread 26430
@ 0x7f2220793422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f2220799c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f222082f4ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 08:05:25.275954 21084 tablet_server.cc:178] TabletServer@127.20.151.1:40455 shutting down...
W20240501 08:05:25.292394 25822 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
I20240501 08:05:25.293152 21084 tablet_copy_service.cc:362] P a8bdf0eecb024a989a57ea4a16594503: Destroying tablet copy session 594bfa72576e4461aa048eb7474768fa-d8aa10ca2f384362841703c074e8ea3b due to service shutdown
I20240501 08:05:25.293613 21084 tablet_copy_service.cc:433] P a8bdf0eecb024a989a57ea4a16594503: ending tablet copy session 594bfa72576e4461aa048eb7474768fa-d8aa10ca2f384362841703c074e8ea3b on tablet d8aa10ca2f384362841703c074e8ea3b with peer 594bfa72576e4461aa048eb7474768fa
@ 0x7f22208603cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f22208461ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f222084617a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
W20240501 08:05:25.372714 26284 consensus_peers.cc:485] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a -> Peer a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455): Couldn't send request to peer a8bdf0eecb024a989a57ea4a16594503. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:25.373463 25822 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.151.1:40455, user_credentials={real_user=slave}} blocked reactor thread for 81087.5us
W20240501 08:05:25.391577 26284 consensus_peers.cc:485] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a -> Peer a8bdf0eecb024a989a57ea4a16594503 (127.20.151.1:40455): Couldn't send request to peer a8bdf0eecb024a989a57ea4a16594503. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:25.392462 25828 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:05:25.392829 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 08:05:25.404114 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:05:25.408378 21084 tablet_replica.cc:330] stopping tablet replica
@ 0x7f222084610a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20240501 08:05:25.409483 21084 raft_consensus.cc:2229] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:05:25.410631 21084 raft_consensus.cc:2258] T c22bf70d91434f688f21d7a288a2d355 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:25.413638 21084 tablet_replica.cc:330] stopping tablet replica
I20240501 08:05:25.420240 21084 raft_consensus.cc:2229] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:25.420790 21084 raft_consensus.cc:2258] T b049db705b3242ecbd1b6483bfee9813 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:25.423452 21084 tablet_replica.cc:330] stopping tablet replica
I20240501 08:05:25.424086 21084 raft_consensus.cc:2229] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:05:25.425341 21084 raft_consensus.cc:2258] T d8aa10ca2f384362841703c074e8ea3b P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:25.428318 21084 tablet_replica.cc:330] stopping tablet replica
I20240501 08:05:25.428921 21084 raft_consensus.cc:2229] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:25.429461 21084 raft_consensus.cc:2258] T 1b9e3f16ca2242069311a4aa01e1a9d3 P a8bdf0eecb024a989a57ea4a16594503 [term 2 FOLLOWER]: Raft consensus is shut down!
@ 0x7f22208460d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f22208452fe std::__1::__function::__func<>::operator()() at ??:0
I20240501 08:05:25.487306 21084 tablet_server.cc:194] TabletServer@127.20.151.1:40455 shutdown complete.
I20240501 08:05:25.523713 21084 tablet_server.cc:178] TabletServer@127.20.151.2:39107 shutting down...
W20240501 08:05:25.533646 25828 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:05:25.534029 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:05:25.538571 25828 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:05:25.539057 25828 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
@ 0x7f2221bebda5 std::__1::__function::__value_func<>::operator()() at ??:0
I20240501 08:05:25.569941 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:05:25.570861 21084 tablet_replica.cc:330] stopping tablet replica
I20240501 08:05:25.571552 21084 raft_consensus.cc:2229] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:25.572120 21084 raft_consensus.cc:2258] T c22bf70d91434f688f21d7a288a2d355 P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:25.574882 21084 tablet_replica.cc:330] stopping tablet replica
I20240501 08:05:25.575732 21084 raft_consensus.cc:2229] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:05:25.576898 21084 raft_consensus.cc:2258] T b049db705b3242ecbd1b6483bfee9813 P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:25.579854 21084 tablet_replica.cc:330] stopping tablet replica
I20240501 08:05:25.580638 21084 raft_consensus.cc:2229] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:25.581250 21084 raft_consensus.cc:2258] T d8aa10ca2f384362841703c074e8ea3b P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:25.583914 21084 tablet_replica.cc:330] stopping tablet replica
I20240501 08:05:25.584638 21084 raft_consensus.cc:2229] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:05:25.585822 21084 raft_consensus.cc:2258] T 1b9e3f16ca2242069311a4aa01e1a9d3 P 412c21db44284526856da410d118cd9a [term 2 FOLLOWER]: Raft consensus is shut down!
@ 0x7f2221bebbda std::__1::function<>::operator()() at ??:0
I20240501 08:05:25.608515 21084 tablet_server.cc:194] TabletServer@127.20.151.2:39107 shutdown complete.
@ 0x7f2218977237 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 08:05:25.630545 21084 tablet_server.cc:178] TabletServer@127.20.151.3:45489 shutting down...
@ 0x7f221897a692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f221897a64a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
I20240501 08:05:25.684707 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:05:25.685576 21084 tablet_replica.cc:330] stopping tablet replica
I20240501 08:05:25.686442 21084 raft_consensus.cc:2229] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:25.687041 21084 raft_consensus.cc:2258] T c22bf70d91434f688f21d7a288a2d355 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:25.690104 21084 tablet_replica.cc:330] stopping tablet replica
I20240501 08:05:25.690852 21084 raft_consensus.cc:2229] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:25.691565 21084 raft_consensus.cc:2258] T b049db705b3242ecbd1b6483bfee9813 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Raft consensus is shut down!
@ 0x7f221897a5da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20240501 08:05:25.694464 21084 tablet_replica.cc:330] stopping tablet replica
I20240501 08:05:25.695214 21084 raft_consensus.cc:2229] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:25.695842 21084 raft_consensus.cc:2258] T d8aa10ca2f384362841703c074e8ea3b P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:05:25.698488 21084 tablet_replica.cc:330] stopping tablet replica
I20240501 08:05:25.699083 21084 raft_consensus.cc:2229] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:05:25.699666 21084 raft_consensus.cc:2258] T 1b9e3f16ca2242069311a4aa01e1a9d3 P f1b6af9bbeab4df9a81e15a2965e6650 [term 2 FOLLOWER]: Raft consensus is shut down!
@ 0x7f221897a5a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20240501 08:05:25.727057 21084 tablet_server.cc:194] TabletServer@127.20.151.3:45489 shutdown complete.
@ 0x7f221897989e std::__1::__function::__func<>::operator()() at ??:0
I20240501 08:05:25.748212 21084 tablet_server.cc:178] TabletServer@127.20.151.4:0 shutting down...
@ 0x7f2221bebda5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7f2221bebbda std::__1::function<>::operator()() at ??:0
I20240501 08:05:25.801182 21084 ts_tablet_manager.cc:1470] Shutting down tablet manager...
@ 0x7f2218959937 kudu::Thread::SuperviseThread() at ??:0