[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240430 17:02:27.292148 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:45401,127.0.103.125:39911,127.0.103.124:46445
I20240430 17:02:27.294061 413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 17:02:27.294997 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:27.313616 419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:27.314270 422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:27.315551 420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:27.315869 413 server_base.cc:1034] running on GCE node
I20240430 17:02:28.469955 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:28.470212 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:28.470410 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496548470384 us; error 0 us; skew 500 ppm
I20240430 17:02:28.471259 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:28.477739 413 webserver.cc:460] Webserver started at http://127.0.103.126:46347/ using document root <none> and password file <none>
I20240430 17:02:28.478526 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:28.478746 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:28.479370 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:28.485842 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-0-root/instance:
uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6"
format_stamp: "Formatted at 2024-04-30 17:02:28 on dist-test-slave-x21r"
I20240430 17:02:28.493876 413 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20240430 17:02:28.499280 428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:28.500463 413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240430 17:02:28.500793 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-0-root
uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6"
format_stamp: "Formatted at 2024-04-30 17:02:28 on dist-test-slave-x21r"
I20240430 17:02:28.501120 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:28.563043 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:28.564554 413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 17:02:28.565001 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:28.644351 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.126:45401 every 8 connection(s)
I20240430 17:02:28.644670 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:45401
I20240430 17:02:28.651407 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240430 17:02:28.652068 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 17:02:28.660353 482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:28.666509 483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:28.671504 485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:28.672554 413 server_base.cc:1034] running on GCE node
I20240430 17:02:28.673195 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:28.673389 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:28.673552 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496548673535 us; error 0 us; skew 500 ppm
I20240430 17:02:28.674075 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:28.677598 413 webserver.cc:460] Webserver started at http://127.0.103.125:34589/ using document root <none> and password file <none>
I20240430 17:02:28.678527 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:28.678715 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:28.678969 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:28.680016 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-1-root/instance:
uuid: "8d478184b78a42ddbe5d25e4e79ef5f5"
format_stamp: "Formatted at 2024-04-30 17:02:28 on dist-test-slave-x21r"
I20240430 17:02:28.685660 413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240430 17:02:28.683254 480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } has no permanent_uuid. Determining permanent_uuid...
I20240430 17:02:28.694985 492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:28.695876 413 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.001s
I20240430 17:02:28.696208 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-1-root
uuid: "8d478184b78a42ddbe5d25e4e79ef5f5"
format_stamp: "Formatted at 2024-04-30 17:02:28 on dist-test-slave-x21r"
I20240430 17:02:28.696588 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:28.713703 480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } has no permanent_uuid. Determining permanent_uuid...
W20240430 17:02:28.716104 430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.125:39911: connect: Connection refused (error 111)
W20240430 17:02:28.720707 480 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.103.125:39911: Network error: Client connection negotiation failed: client connection to 127.0.103.125:39911: connect: Connection refused (error 111)
I20240430 17:02:28.722262 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:28.723484 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:28.770259 480 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } attempt: 1
W20240430 17:02:28.775858 480 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.103.125:39911: Network error: Client connection negotiation failed: client connection to 127.0.103.125:39911: connect: Connection refused (error 111)
I20240430 17:02:28.797179 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.125:39911
I20240430 17:02:28.797256 544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.125:39911 every 8 connection(s)
I20240430 17:02:28.802142 545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:28.803042 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240430 17:02:28.807716 545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } has no permanent_uuid. Determining permanent_uuid...
W20240430 17:02:28.811707 547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:28.815701 548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:28.825143 551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:28.826174 413 server_base.cc:1034] running on GCE node
I20240430 17:02:28.827064 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:28.827334 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:28.827505 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496548827485 us; error 0 us; skew 500 ppm
I20240430 17:02:28.828214 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:28.828307 545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } has no permanent_uuid. Determining permanent_uuid...
I20240430 17:02:28.832417 413 webserver.cc:460] Webserver started at http://127.0.103.124:39197/ using document root <none> and password file <none>
I20240430 17:02:28.832902 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:28.833089 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:28.833361 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:28.834538 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-2-root/instance:
uuid: "08f0226936e04064ac84ba881fbfa8a4"
format_stamp: "Formatted at 2024-04-30 17:02:28 on dist-test-slave-x21r"
I20240430 17:02:28.840108 413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240430 17:02:28.841835 545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } has no permanent_uuid. Determining permanent_uuid...
I20240430 17:02:28.844446 480 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } attempt: 2
I20240430 17:02:28.845326 557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:28.846534 413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240430 17:02:28.846890 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-2-root
uuid: "08f0226936e04064ac84ba881fbfa8a4"
format_stamp: "Formatted at 2024-04-30 17:02:28 on dist-test-slave-x21r"
I20240430 17:02:28.847297 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240430 17:02:28.848731 545 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.103.124:46445: Network error: Client connection negotiation failed: client connection to 127.0.103.124:46445: connect: Connection refused (error 111)
I20240430 17:02:28.858444 480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } has no permanent_uuid. Determining permanent_uuid...
W20240430 17:02:28.862393 480 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.103.124:46445: Network error: Client connection negotiation failed: client connection to 127.0.103.124:46445: connect: Connection refused (error 111)
I20240430 17:02:28.864840 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:28.866364 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:28.892481 545 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } attempt: 1
I20240430 17:02:28.893864 480 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } attempt: 1
W20240430 17:02:28.899335 545 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.103.124:46445: Network error: Client connection negotiation failed: client connection to 127.0.103.124:46445: connect: Connection refused (error 111)
W20240430 17:02:28.899881 480 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.103.124:46445: Network error: Client connection negotiation failed: client connection to 127.0.103.124:46445: connect: Connection refused (error 111)
I20240430 17:02:28.946400 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.124:46445
I20240430 17:02:28.946503 609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.124:46445 every 8 connection(s)
I20240430 17:02:28.951458 610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:28.951436 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240430 17:02:28.957115 610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } has no permanent_uuid. Determining permanent_uuid...
I20240430 17:02:28.967411 480 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } attempt: 2
I20240430 17:02:28.969542 610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } has no permanent_uuid. Determining permanent_uuid...
I20240430 17:02:28.974849 545 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } attempt: 2
I20240430 17:02:28.981200 610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } has no permanent_uuid. Determining permanent_uuid...
I20240430 17:02:28.989632 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6: Bootstrap starting.
I20240430 17:02:28.997609 545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5: Bootstrap starting.
I20240430 17:02:29.000185 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:29.002686 480 log.cc:824] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6: Log is configured to *not* fsync() on all Append() calls
I20240430 17:02:29.002961 610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4: Bootstrap starting.
I20240430 17:02:29.004493 545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:29.008174 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6: No bootstrap required, opened a new log
I20240430 17:02:29.009258 610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:29.013268 545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5: No bootstrap required, opened a new log
I20240430 17:02:29.015190 610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4: No bootstrap required, opened a new log
I20240430 17:02:29.032423 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } }
I20240430 17:02:29.032339 545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } }
I20240430 17:02:29.032339 610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } }
I20240430 17:02:29.033205 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:29.033376 610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:29.033344 545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:29.033569 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5712f4a8e86b4d7da5d8f2eeb6b6a0d6, State: Initialized, Role: FOLLOWER
I20240430 17:02:29.033742 610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f0226936e04064ac84ba881fbfa8a4, State: Initialized, Role: FOLLOWER
I20240430 17:02:29.033792 545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d478184b78a42ddbe5d25e4e79ef5f5, State: Initialized, Role: FOLLOWER
I20240430 17:02:29.034557 610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } }
I20240430 17:02:29.034564 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } }
I20240430 17:02:29.034780 545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } }
I20240430 17:02:29.040597 620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } } }
I20240430 17:02:29.041846 620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:29.043366 545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 17:02:29.043138 619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } } }
I20240430 17:02:29.044179 619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:29.045318 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 17:02:29.046098 618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } } }
I20240430 17:02:29.047025 618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:29.048233 610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [sys.catalog]: configured and running, proceeding with master startup.
W20240430 17:02:29.072096 434 tablet.cc:2369] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 17:02:29.085515 618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:29.086266 618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } }
I20240430 17:02:29.092259 618 leader_election.cc:288] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 (127.0.103.126:45401), 8d478184b78a42ddbe5d25e4e79ef5f5 (127.0.103.125:39911)
I20240430 17:02:29.092083 455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08f0226936e04064ac84ba881fbfa8a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" is_pre_election: true
I20240430 17:02:29.092911 455 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08f0226936e04064ac84ba881fbfa8a4 in term 0.
I20240430 17:02:29.094070 561 leader_election.cc:302] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08f0226936e04064ac84ba881fbfa8a4, 5712f4a8e86b4d7da5d8f2eeb6b6a0d6; no voters:
I20240430 17:02:29.093935 520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08f0226936e04064ac84ba881fbfa8a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" is_pre_election: true
I20240430 17:02:29.094913 520 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08f0226936e04064ac84ba881fbfa8a4 in term 0.
I20240430 17:02:29.095006 618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:02:29.096619 618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:29.096966 618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:29.106787 649 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240430 17:02:29.111896 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240430 17:02:29.112190 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240430 17:02:29.118113 652 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240430 17:02:29.118371 652 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240430 17:02:29.178159 618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } }
I20240430 17:02:29.181463 455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08f0226936e04064ac84ba881fbfa8a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6"
I20240430 17:02:29.182144 455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:29.186805 618 leader_election.cc:288] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [CANDIDATE]: Term 1 election: Requested vote from peers 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 (127.0.103.126:45401), 8d478184b78a42ddbe5d25e4e79ef5f5 (127.0.103.125:39911)
I20240430 17:02:29.188350 520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08f0226936e04064ac84ba881fbfa8a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5"
I20240430 17:02:29.188947 520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:29.192159 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:29.195400 654 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240430 17:02:29.195629 654 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240430 17:02:29.201635 655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:29.205711 656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:29.210187 658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:29.211475 413 server_base.cc:1034] running on GCE node
I20240430 17:02:29.212164 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:29.212365 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:29.212528 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496549212509 us; error 0 us; skew 500 ppm
I20240430 17:02:29.213023 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:29.216415 413 webserver.cc:460] Webserver started at http://127.0.103.65:35943/ using document root <none> and password file <none>
I20240430 17:02:29.216853 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:29.217028 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:29.217288 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:29.218354 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-0-root/instance:
uuid: "e573360eaf7b4aeba7d5b9ee9f04f017"
format_stamp: "Formatted at 2024-04-30 17:02:29 on dist-test-slave-x21r"
I20240430 17:02:29.223008 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240430 17:02:29.226503 663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:29.227345 413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240430 17:02:29.227599 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-0-root
uuid: "e573360eaf7b4aeba7d5b9ee9f04f017"
format_stamp: "Formatted at 2024-04-30 17:02:29 on dist-test-slave-x21r"
I20240430 17:02:29.227854 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:29.247421 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:29.248968 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:29.251614 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:29.255728 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:29.256033 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:29.256292 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:29.256449 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:29.269116 455 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08f0226936e04064ac84ba881fbfa8a4 in term 1.
I20240430 17:02:29.270871 561 leader_election.cc:302] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08f0226936e04064ac84ba881fbfa8a4, 5712f4a8e86b4d7da5d8f2eeb6b6a0d6; no voters:
I20240430 17:02:29.272310 618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:29.276043 520 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08f0226936e04064ac84ba881fbfa8a4 in term 1.
I20240430 17:02:29.278215 618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 1 LEADER]: Becoming Leader. State: Replica: 08f0226936e04064ac84ba881fbfa8a4, State: Running, Role: LEADER
I20240430 17:02:29.279999 618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } }
I20240430 17:02:29.291457 673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08f0226936e04064ac84ba881fbfa8a4. Latest consensus state: current_term: 1 leader_uuid: "08f0226936e04064ac84ba881fbfa8a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } } }
I20240430 17:02:29.293006 673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [sys.catalog]: This master's current role is: LEADER
I20240430 17:02:29.297572 680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 17:02:29.306303 680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 17:02:29.331337 520 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [term 1 FOLLOWER]: Refusing update from remote peer 08f0226936e04064ac84ba881fbfa8a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:02:29.331856 455 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [term 1 FOLLOWER]: Refusing update from remote peer 08f0226936e04064ac84ba881fbfa8a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:02:29.333714 673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240430 17:02:29.336967 618 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:29.358784 620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08f0226936e04064ac84ba881fbfa8a4. Latest consensus state: current_term: 1 leader_uuid: "08f0226936e04064ac84ba881fbfa8a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } } }
I20240430 17:02:29.360591 620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:29.369722 692 mvcc.cc:205] Tried to move back new op lower bound from 7022577866030583808 to 7022577865865670656. Current Snapshot: MvccSnapshot[applied={T|T < 7022577866030583808}]
I20240430 17:02:29.375790 620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08f0226936e04064ac84ba881fbfa8a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } } }
I20240430 17:02:29.377089 620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:29.384148 619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08f0226936e04064ac84ba881fbfa8a4. Latest consensus state: current_term: 1 leader_uuid: "08f0226936e04064ac84ba881fbfa8a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } } }
I20240430 17:02:29.385557 619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:29.387845 673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08f0226936e04064ac84ba881fbfa8a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } } }
I20240430 17:02:29.391129 673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [sys.catalog]: This master's current role is: LEADER
I20240430 17:02:29.396327 680 catalog_manager.cc:1347] Generated new cluster ID: bfae936167ca42398736a0c818d6d0ff
I20240430 17:02:29.396659 680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 17:02:29.396486 673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08f0226936e04064ac84ba881fbfa8a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } } }
I20240430 17:02:29.399201 673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [sys.catalog]: This master's current role is: LEADER
I20240430 17:02:29.404008 619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08f0226936e04064ac84ba881fbfa8a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5712f4a8e86b4d7da5d8f2eeb6b6a0d6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45401 } } peers { permanent_uuid: "8d478184b78a42ddbe5d25e4e79ef5f5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 39911 } } peers { permanent_uuid: "08f0226936e04064ac84ba881fbfa8a4" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 46445 } } }
I20240430 17:02:29.405301 619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:29.444407 680 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 17:02:29.446557 680 catalog_manager.cc:1504] Loading token signing keys...
I20240430 17:02:29.491583 680 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4: Generated new TSK 0
I20240430 17:02:29.492812 680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 17:02:29.507905 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:45553
I20240430 17:02:29.508069 737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.65:45553 every 8 connection(s)
I20240430 17:02:29.541864 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240430 17:02:29.549763 740 heartbeater.cc:347] Connected to a master server at 127.0.103.126:45401
I20240430 17:02:29.550240 740 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:29.551458 740 heartbeater.cc:511] Master 127.0.103.126:45401 requested a full tablet report, sending...
I20240430 17:02:29.553892 739 heartbeater.cc:347] Connected to a master server at 127.0.103.125:39911
I20240430 17:02:29.554381 739 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:29.556430 739 heartbeater.cc:511] Master 127.0.103.125:39911 requested a full tablet report, sending...
I20240430 17:02:29.557992 445 ts_manager.cc:194] Registered new tserver with Master: e573360eaf7b4aeba7d5b9ee9f04f017 (127.0.103.65:45553)
I20240430 17:02:29.561640 510 ts_manager.cc:194] Registered new tserver with Master: e573360eaf7b4aeba7d5b9ee9f04f017 (127.0.103.65:45553)
I20240430 17:02:29.562177 738 heartbeater.cc:347] Connected to a master server at 127.0.103.124:46445
I20240430 17:02:29.562654 738 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:29.563534 738 heartbeater.cc:511] Master 127.0.103.124:46445 requested a full tablet report, sending...
I20240430 17:02:29.566452 575 ts_manager.cc:194] Registered new tserver with Master: e573360eaf7b4aeba7d5b9ee9f04f017 (127.0.103.65:45553)
W20240430 17:02:29.569780 748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:29.572860 575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46282
W20240430 17:02:29.575906 751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:29.576794 749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:29.577754 413 server_base.cc:1034] running on GCE node
I20240430 17:02:29.578557 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:29.578750 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:29.578920 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496549578901 us; error 0 us; skew 500 ppm
I20240430 17:02:29.579435 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:29.582309 413 webserver.cc:460] Webserver started at http://127.0.103.66:36589/ using document root <none> and password file <none>
I20240430 17:02:29.582765 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:29.582939 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:29.583220 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:29.584236 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-1-root/instance:
uuid: "3170f14253ab4e16adce8fb41e096299"
format_stamp: "Formatted at 2024-04-30 17:02:29 on dist-test-slave-x21r"
I20240430 17:02:29.588624 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240430 17:02:29.591702 756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:29.592478 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240430 17:02:29.592727 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-1-root
uuid: "3170f14253ab4e16adce8fb41e096299"
format_stamp: "Formatted at 2024-04-30 17:02:29 on dist-test-slave-x21r"
I20240430 17:02:29.592974 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:29.614362 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:29.615554 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:29.616979 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:29.619325 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:29.619511 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:29.619755 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:29.619904 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:29.733340 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:38873
I20240430 17:02:29.733422 818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.66:38873 every 8 connection(s)
I20240430 17:02:29.771806 820 heartbeater.cc:347] Connected to a master server at 127.0.103.125:39911
I20240430 17:02:29.772255 820 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:29.773314 820 heartbeater.cc:511] Master 127.0.103.125:39911 requested a full tablet report, sending...
I20240430 17:02:29.774991 819 heartbeater.cc:347] Connected to a master server at 127.0.103.124:46445
I20240430 17:02:29.775532 819 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:29.775993 510 ts_manager.cc:194] Registered new tserver with Master: 3170f14253ab4e16adce8fb41e096299 (127.0.103.66:38873)
I20240430 17:02:29.779743 819 heartbeater.cc:511] Master 127.0.103.124:46445 requested a full tablet report, sending...
I20240430 17:02:29.780714 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240430 17:02:29.784363 575 ts_manager.cc:194] Registered new tserver with Master: 3170f14253ab4e16adce8fb41e096299 (127.0.103.66:38873)
I20240430 17:02:29.786131 575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59668
W20240430 17:02:29.797832 826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:29.798338 822 heartbeater.cc:347] Connected to a master server at 127.0.103.126:45401
I20240430 17:02:29.798794 822 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:29.799583 822 heartbeater.cc:511] Master 127.0.103.126:45401 requested a full tablet report, sending...
W20240430 17:02:29.802179 827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:29.802232 445 ts_manager.cc:194] Registered new tserver with Master: 3170f14253ab4e16adce8fb41e096299 (127.0.103.66:38873)
I20240430 17:02:29.808589 413 server_base.cc:1034] running on GCE node
W20240430 17:02:29.809453 829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:29.810310 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:29.810545 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:29.810731 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496549810714 us; error 0 us; skew 500 ppm
I20240430 17:02:29.811273 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:29.814162 413 webserver.cc:460] Webserver started at http://127.0.103.67:33563/ using document root <none> and password file <none>
I20240430 17:02:29.814599 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:29.814781 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:29.815025 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:29.816076 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-2-root/instance:
uuid: "43434212f851415ba183d46ac46bcf56"
format_stamp: "Formatted at 2024-04-30 17:02:29 on dist-test-slave-x21r"
I20240430 17:02:29.820448 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240430 17:02:29.823854 834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:29.824673 413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240430 17:02:29.824936 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-2-root
uuid: "43434212f851415ba183d46ac46bcf56"
format_stamp: "Formatted at 2024-04-30 17:02:29 on dist-test-slave-x21r"
I20240430 17:02:29.825248 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:29.846278 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:29.847525 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:29.850212 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:29.854279 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:29.854487 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:29.854712 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:29.854864 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:29.965448 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:41953
I20240430 17:02:29.965732 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.67:41953 every 8 connection(s)
I20240430 17:02:30.008929 897 heartbeater.cc:347] Connected to a master server at 127.0.103.124:46445
I20240430 17:02:30.009341 897 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:30.010316 897 heartbeater.cc:511] Master 127.0.103.124:46445 requested a full tablet report, sending...
I20240430 17:02:30.010994 898 heartbeater.cc:347] Connected to a master server at 127.0.103.125:39911
I20240430 17:02:30.011441 898 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:30.012354 898 heartbeater.cc:511] Master 127.0.103.125:39911 requested a full tablet report, sending...
I20240430 17:02:30.012506 575 ts_manager.cc:194] Registered new tserver with Master: 43434212f851415ba183d46ac46bcf56 (127.0.103.67:41953)
I20240430 17:02:30.014391 575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59676
I20240430 17:02:30.015403 510 ts_manager.cc:194] Registered new tserver with Master: 43434212f851415ba183d46ac46bcf56 (127.0.103.67:41953)
I20240430 17:02:30.017822 900 heartbeater.cc:347] Connected to a master server at 127.0.103.126:45401
I20240430 17:02:30.018131 900 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:30.018661 900 heartbeater.cc:511] Master 127.0.103.126:45401 requested a full tablet report, sending...
I20240430 17:02:30.020764 445 ts_manager.cc:194] Registered new tserver with Master: 43434212f851415ba183d46ac46bcf56 (127.0.103.67:41953)
I20240430 17:02:30.021211 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013129855s
I20240430 17:02:30.111348 649 catalog_manager.cc:1259] Loaded cluster ID: bfae936167ca42398736a0c818d6d0ff
I20240430 17:02:30.111629 649 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5: loading cluster ID for follower catalog manager: success
I20240430 17:02:30.116086 649 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5: acquiring CA information for follower catalog manager: success
I20240430 17:02:30.121043 649 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240430 17:02:30.122726 652 catalog_manager.cc:1259] Loaded cluster ID: bfae936167ca42398736a0c818d6d0ff
I20240430 17:02:30.122972 652 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6: loading cluster ID for follower catalog manager: success
I20240430 17:02:30.127641 652 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6: acquiring CA information for follower catalog manager: success
I20240430 17:02:30.130913 652 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240430 17:02:30.577270 738 heartbeater.cc:503] Master 127.0.103.124:46445 was elected leader, sending a full tablet report...
I20240430 17:02:30.790902 819 heartbeater.cc:503] Master 127.0.103.124:46445 was elected leader, sending a full tablet report...
I20240430 17:02:31.016971 897 heartbeater.cc:503] Master 127.0.103.124:46445 was elected leader, sending a full tablet report...
I20240430 17:02:31.107467 645 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:02:31.135318 413 test_util.cc:268] Using random seed: 1441146937
I20240430 17:02:31.212733 575 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59688:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 17:02:31.328990 784 tablet_service.cc:1467] Processing CreateTablet for tablet 49a08bc30d844c3390456af86c98ecba (DEFAULT_TABLE table=test-workload [id=576e583de75a402c80bbd44741168e59]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240430 17:02:31.329663 695 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8db2b6225b492b8e29bcae60e36a78 (DEFAULT_TABLE table=test-workload [id=576e583de75a402c80bbd44741168e59]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240430 17:02:31.330117 694 tablet_service.cc:1467] Processing CreateTablet for tablet 505970c1b47c447aa8382926f1480cfa (DEFAULT_TABLE table=test-workload [id=576e583de75a402c80bbd44741168e59]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240430 17:02:31.331172 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8db2b6225b492b8e29bcae60e36a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:31.331750 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 505970c1b47c447aa8382926f1480cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:31.332799 862 tablet_service.cc:1467] Processing CreateTablet for tablet d28dd0dd8b6342238d82b3cd413b9c7e (DEFAULT_TABLE table=test-workload [id=576e583de75a402c80bbd44741168e59]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240430 17:02:31.334061 862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d28dd0dd8b6342238d82b3cd413b9c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:31.339541 784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a08bc30d844c3390456af86c98ecba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:31.348515 933 tablet_bootstrap.cc:492] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017: Bootstrap starting.
I20240430 17:02:31.353760 933 tablet_bootstrap.cc:654] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:31.359598 934 tablet_bootstrap.cc:492] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56: Bootstrap starting.
I20240430 17:02:31.365813 935 tablet_bootstrap.cc:492] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299: Bootstrap starting.
I20240430 17:02:31.366782 933 tablet_bootstrap.cc:492] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017: No bootstrap required, opened a new log
I20240430 17:02:31.367220 933 ts_tablet_manager.cc:1367] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.009s
I20240430 17:02:31.368721 934 tablet_bootstrap.cc:654] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:31.369691 933 raft_consensus.cc:348] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e573360eaf7b4aeba7d5b9ee9f04f017" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45553 } }
I20240430 17:02:31.370663 933 raft_consensus.cc:374] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:31.370931 933 raft_consensus.cc:729] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e573360eaf7b4aeba7d5b9ee9f04f017, State: Initialized, Role: FOLLOWER
I20240430 17:02:31.371678 933 consensus_queue.cc:260] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e573360eaf7b4aeba7d5b9ee9f04f017" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45553 } }
I20240430 17:02:31.372368 933 raft_consensus.cc:388] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:02:31.372723 933 raft_consensus.cc:482] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:02:31.373055 933 raft_consensus.cc:3037] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:31.373214 935 tablet_bootstrap.cc:654] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:31.377895 934 tablet_bootstrap.cc:492] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56: No bootstrap required, opened a new log
I20240430 17:02:31.378343 934 ts_tablet_manager.cc:1367] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.012s
I20240430 17:02:31.379238 933 raft_consensus.cc:504] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e573360eaf7b4aeba7d5b9ee9f04f017" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45553 } }
I20240430 17:02:31.379779 935 tablet_bootstrap.cc:492] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299: No bootstrap required, opened a new log
I20240430 17:02:31.379797 933 leader_election.cc:302] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e573360eaf7b4aeba7d5b9ee9f04f017; no voters:
I20240430 17:02:31.380492 935 ts_tablet_manager.cc:1367] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240430 17:02:31.380883 934 raft_consensus.cc:348] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43434212f851415ba183d46ac46bcf56" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41953 } }
I20240430 17:02:31.381465 934 raft_consensus.cc:374] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:31.381660 934 raft_consensus.cc:729] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43434212f851415ba183d46ac46bcf56, State: Initialized, Role: FOLLOWER
I20240430 17:02:31.382227 934 consensus_queue.cc:260] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43434212f851415ba183d46ac46bcf56" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41953 } }
I20240430 17:02:31.382782 933 leader_election.cc:288] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:02:31.383204 934 raft_consensus.cc:388] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:02:31.383546 939 raft_consensus.cc:2781] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:31.383569 934 raft_consensus.cc:482] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:02:31.384321 934 raft_consensus.cc:3037] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:31.384147 935 raft_consensus.cc:348] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3170f14253ab4e16adce8fb41e096299" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38873 } }
I20240430 17:02:31.384748 935 raft_consensus.cc:374] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:31.385072 935 raft_consensus.cc:729] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3170f14253ab4e16adce8fb41e096299, State: Initialized, Role: FOLLOWER
I20240430 17:02:31.385819 935 consensus_queue.cc:260] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3170f14253ab4e16adce8fb41e096299" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38873 } }
I20240430 17:02:31.386462 935 raft_consensus.cc:388] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:02:31.386775 935 raft_consensus.cc:482] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:02:31.387182 935 raft_consensus.cc:3037] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:31.391347 934 raft_consensus.cc:504] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43434212f851415ba183d46ac46bcf56" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41953 } }
I20240430 17:02:31.392082 934 leader_election.cc:302] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43434212f851415ba183d46ac46bcf56; no voters:
I20240430 17:02:31.393294 935 raft_consensus.cc:504] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3170f14253ab4e16adce8fb41e096299" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38873 } }
I20240430 17:02:31.394413 935 leader_election.cc:302] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3170f14253ab4e16adce8fb41e096299; no voters:
I20240430 17:02:31.396029 933 ts_tablet_manager.cc:1398] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017: Time spent starting tablet: real 0.029s user 0.019s sys 0.008s
I20240430 17:02:31.394821 939 raft_consensus.cc:686] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [term 1 LEADER]: Becoming Leader. State: Replica: e573360eaf7b4aeba7d5b9ee9f04f017, State: Running, Role: LEADER
I20240430 17:02:31.397516 933 tablet_bootstrap.cc:492] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017: Bootstrap starting.
I20240430 17:02:31.398454 939 consensus_queue.cc:237] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e573360eaf7b4aeba7d5b9ee9f04f017" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45553 } }
I20240430 17:02:31.399024 943 raft_consensus.cc:2781] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:31.400540 935 leader_election.cc:288] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:02:31.400540 934 leader_election.cc:288] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:02:31.406673 942 raft_consensus.cc:2781] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:31.409320 933 tablet_bootstrap.cc:654] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:31.415939 934 ts_tablet_manager.cc:1398] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56: Time spent starting tablet: real 0.037s user 0.008s sys 0.018s
I20240430 17:02:31.425226 943 raft_consensus.cc:686] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [term 1 LEADER]: Becoming Leader. State: Replica: 3170f14253ab4e16adce8fb41e096299, State: Running, Role: LEADER
I20240430 17:02:31.426949 943 consensus_queue.cc:237] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3170f14253ab4e16adce8fb41e096299" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38873 } }
I20240430 17:02:31.431084 942 raft_consensus.cc:686] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [term 1 LEADER]: Becoming Leader. State: Replica: 43434212f851415ba183d46ac46bcf56, State: Running, Role: LEADER
I20240430 17:02:31.432832 942 consensus_queue.cc:237] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43434212f851415ba183d46ac46bcf56" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41953 } }
I20240430 17:02:31.434751 575 catalog_manager.cc:5484] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 reported cstate change: term changed from 0 to 1, leader changed from <none> to e573360eaf7b4aeba7d5b9ee9f04f017 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e573360eaf7b4aeba7d5b9ee9f04f017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e573360eaf7b4aeba7d5b9ee9f04f017" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45553 } health_report { overall_health: HEALTHY } } }
I20240430 17:02:31.444835 935 ts_tablet_manager.cc:1398] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299: Time spent starting tablet: real 0.064s user 0.019s sys 0.009s
I20240430 17:02:31.458120 574 catalog_manager.cc:5484] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3170f14253ab4e16adce8fb41e096299 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "3170f14253ab4e16adce8fb41e096299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3170f14253ab4e16adce8fb41e096299" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38873 } health_report { overall_health: HEALTHY } } }
I20240430 17:02:31.457377 933 tablet_bootstrap.cc:492] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017: No bootstrap required, opened a new log
I20240430 17:02:31.461798 933 ts_tablet_manager.cc:1367] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.008s
I20240430 17:02:31.500716 933 raft_consensus.cc:348] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e573360eaf7b4aeba7d5b9ee9f04f017" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45553 } }
I20240430 17:02:31.501928 933 raft_consensus.cc:374] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:31.502240 933 raft_consensus.cc:729] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e573360eaf7b4aeba7d5b9ee9f04f017, State: Initialized, Role: FOLLOWER
I20240430 17:02:31.502941 933 consensus_queue.cc:260] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e573360eaf7b4aeba7d5b9ee9f04f017" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45553 } }
I20240430 17:02:31.503626 933 raft_consensus.cc:388] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:02:31.505841 933 raft_consensus.cc:482] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:02:31.506181 933 raft_consensus.cc:3037] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:31.507894 573 catalog_manager.cc:5484] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43434212f851415ba183d46ac46bcf56 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "43434212f851415ba183d46ac46bcf56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43434212f851415ba183d46ac46bcf56" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41953 } health_report { overall_health: HEALTHY } } }
I20240430 17:02:31.515735 933 raft_consensus.cc:504] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e573360eaf7b4aeba7d5b9ee9f04f017" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45553 } }
I20240430 17:02:31.519407 933 leader_election.cc:302] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e573360eaf7b4aeba7d5b9ee9f04f017; no voters:
I20240430 17:02:31.525537 933 leader_election.cc:288] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:02:31.526577 940 raft_consensus.cc:2781] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:31.527079 940 raft_consensus.cc:686] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [term 1 LEADER]: Becoming Leader. State: Replica: e573360eaf7b4aeba7d5b9ee9f04f017, State: Running, Role: LEADER
I20240430 17:02:31.527786 940 consensus_queue.cc:237] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e573360eaf7b4aeba7d5b9ee9f04f017" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45553 } }
I20240430 17:02:31.532289 933 ts_tablet_manager.cc:1398] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017: Time spent starting tablet: real 0.065s user 0.010s sys 0.008s
I20240430 17:02:31.559288 574 catalog_manager.cc:5484] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 reported cstate change: term changed from 0 to 1, leader changed from <none> to e573360eaf7b4aeba7d5b9ee9f04f017 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e573360eaf7b4aeba7d5b9ee9f04f017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e573360eaf7b4aeba7d5b9ee9f04f017" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45553 } health_report { overall_health: HEALTHY } } }
I20240430 17:02:31.601424 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20240430 17:02:31.618140 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:02:31.618988 413 tablet_replica.cc:330] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017: stopping tablet replica
I20240430 17:02:31.619706 413 raft_consensus.cc:2229] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:31.620168 413 raft_consensus.cc:2258] T 6c8db2b6225b492b8e29bcae60e36a78 P e573360eaf7b4aeba7d5b9ee9f04f017 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:31.622629 413 tablet_replica.cc:330] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017: stopping tablet replica
I20240430 17:02:31.623064 413 raft_consensus.cc:2229] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:31.623534 413 raft_consensus.cc:2258] T 505970c1b47c447aa8382926f1480cfa P e573360eaf7b4aeba7d5b9ee9f04f017 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:31.777179 413 tablet_server.cc:194] TabletServer@127.0.103.65:0 shutdown complete.
I20240430 17:02:31.786145 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20240430 17:02:31.802623 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:02:31.803340 413 tablet_replica.cc:330] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299: stopping tablet replica
I20240430 17:02:31.803787 413 raft_consensus.cc:2229] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:31.804157 413 raft_consensus.cc:2258] T 49a08bc30d844c3390456af86c98ecba P 3170f14253ab4e16adce8fb41e096299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:31.812510 413 tablet_server.cc:194] TabletServer@127.0.103.66:0 shutdown complete.
I20240430 17:02:31.819727 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20240430 17:02:31.837764 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:02:31.838531 413 tablet_replica.cc:330] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56: stopping tablet replica
I20240430 17:02:31.839020 413 raft_consensus.cc:2229] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:31.839650 413 raft_consensus.cc:2258] T d28dd0dd8b6342238d82b3cd413b9c7e P 43434212f851415ba183d46ac46bcf56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:31.858963 413 tablet_server.cc:194] TabletServer@127.0.103.67:0 shutdown complete.
I20240430 17:02:31.866516 413 master.cc:537] Master@127.0.103.126:45401 shutting down...
I20240430 17:02:31.989486 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:31.989991 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:31.990253 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5712f4a8e86b4d7da5d8f2eeb6b6a0d6: stopping tablet replica
I20240430 17:02:32.016722 413 master.cc:552] Master@127.0.103.126:45401 shutdown complete.
I20240430 17:02:32.051352 413 master.cc:537] Master@127.0.103.125:39911 shutting down...
I20240430 17:02:32.065079 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:32.065531 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:32.065778 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8d478184b78a42ddbe5d25e4e79ef5f5: stopping tablet replica
W20240430 17:02:32.106959 561 consensus_peers.cc:485] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 -> Peer 5712f4a8e86b4d7da5d8f2eeb6b6a0d6 (127.0.103.126:45401): Couldn't send request to peer 5712f4a8e86b4d7da5d8f2eeb6b6a0d6. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.126:45401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240430 17:02:32.112861 560 consensus_peers.cc:485] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 -> Peer 8d478184b78a42ddbe5d25e4e79ef5f5 (127.0.103.125:39911): Couldn't send request to peer 8d478184b78a42ddbe5d25e4e79ef5f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.125:39911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 17:02:32.124796 413 master.cc:552] Master@127.0.103.125:39911 shutdown complete.
I20240430 17:02:32.136713 413 master.cc:537] Master@127.0.103.124:46445 shutting down...
I20240430 17:02:32.151424 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:32.152227 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:32.152546 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P 08f0226936e04064ac84ba881fbfa8a4: stopping tablet replica
I20240430 17:02:32.169930 413 master.cc:552] Master@127.0.103.124:46445 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (4907 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240430 17:02:32.196061 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:40275
I20240430 17:02:32.197072 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:32.202631 958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:32.203373 959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:32.204115 961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:32.204939 413 server_base.cc:1034] running on GCE node
I20240430 17:02:32.205569 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:32.205749 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:32.205891 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496552205874 us; error 0 us; skew 500 ppm
I20240430 17:02:32.206349 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:32.208503 413 webserver.cc:460] Webserver started at http://127.0.103.126:42317/ using document root <none> and password file <none>
I20240430 17:02:32.208949 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:32.209121 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:32.209359 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:32.210347 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/master-0-root/instance:
uuid: "d90eaf52a5e94b91bc5cfe924bc7d017"
format_stamp: "Formatted at 2024-04-30 17:02:32 on dist-test-slave-x21r"
I20240430 17:02:32.214722 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240430 17:02:32.217768 966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:32.218524 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240430 17:02:32.218787 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/master-0-root
uuid: "d90eaf52a5e94b91bc5cfe924bc7d017"
format_stamp: "Formatted at 2024-04-30 17:02:32 on dist-test-slave-x21r"
I20240430 17:02:32.219074 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:32.241806 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:32.242789 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:32.275508 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:40275
I20240430 17:02:32.275605 1017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.126:40275 every 8 connection(s)
I20240430 17:02:32.278941 1018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:32.287647 1018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017: Bootstrap starting.
I20240430 17:02:32.291266 1018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:32.294747 1018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017: No bootstrap required, opened a new log
I20240430 17:02:32.296409 1018 raft_consensus.cc:348] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90eaf52a5e94b91bc5cfe924bc7d017" member_type: VOTER }
I20240430 17:02:32.296772 1018 raft_consensus.cc:374] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:32.296993 1018 raft_consensus.cc:729] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90eaf52a5e94b91bc5cfe924bc7d017, State: Initialized, Role: FOLLOWER
I20240430 17:02:32.297549 1018 consensus_queue.cc:260] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90eaf52a5e94b91bc5cfe924bc7d017" member_type: VOTER }
I20240430 17:02:32.297969 1018 raft_consensus.cc:388] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:02:32.298185 1018 raft_consensus.cc:482] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:02:32.298426 1018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:32.302114 1018 raft_consensus.cc:504] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90eaf52a5e94b91bc5cfe924bc7d017" member_type: VOTER }
I20240430 17:02:32.302587 1018 leader_election.cc:302] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d90eaf52a5e94b91bc5cfe924bc7d017; no voters:
I20240430 17:02:32.303696 1018 leader_election.cc:288] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:02:32.303970 1021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:32.305285 1021 raft_consensus.cc:686] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [term 1 LEADER]: Becoming Leader. State: Replica: d90eaf52a5e94b91bc5cfe924bc7d017, State: Running, Role: LEADER
I20240430 17:02:32.305894 1021 consensus_queue.cc:237] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90eaf52a5e94b91bc5cfe924bc7d017" member_type: VOTER }
I20240430 17:02:32.306355 1018 sys_catalog.cc:564] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 17:02:32.310456 1023 sys_catalog.cc:455] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d90eaf52a5e94b91bc5cfe924bc7d017. Latest consensus state: current_term: 1 leader_uuid: "d90eaf52a5e94b91bc5cfe924bc7d017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90eaf52a5e94b91bc5cfe924bc7d017" member_type: VOTER } }
I20240430 17:02:32.311175 1023 sys_catalog.cc:458] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [sys.catalog]: This master's current role is: LEADER
I20240430 17:02:32.311539 1022 sys_catalog.cc:455] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d90eaf52a5e94b91bc5cfe924bc7d017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90eaf52a5e94b91bc5cfe924bc7d017" member_type: VOTER } }
I20240430 17:02:32.312088 1022 sys_catalog.cc:458] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [sys.catalog]: This master's current role is: LEADER
I20240430 17:02:32.319732 1029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 17:02:32.324412 1029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 17:02:32.326066 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240430 17:02:32.332198 1029 catalog_manager.cc:1347] Generated new cluster ID: 0dccd79d32544c9b815a0ba2d8f32e8a
I20240430 17:02:32.332479 1029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 17:02:32.357445 1029 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 17:02:32.358609 1029 catalog_manager.cc:1504] Loading token signing keys...
I20240430 17:02:32.372176 1029 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017: Generated new TSK 0
I20240430 17:02:32.372769 1029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 17:02:32.392171 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:32.398363 1039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:32.399590 1040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:32.400341 1042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:32.401057 413 server_base.cc:1034] running on GCE node
I20240430 17:02:32.401789 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:32.401974 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:32.402097 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496552402087 us; error 0 us; skew 500 ppm
I20240430 17:02:32.402537 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:32.404789 413 webserver.cc:460] Webserver started at http://127.0.103.65:41711/ using document root <none> and password file <none>
I20240430 17:02:32.405211 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:32.405359 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:32.405563 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:32.406536 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-0-root/instance:
uuid: "214687450caa4ed99c39a919d01bb8ab"
format_stamp: "Formatted at 2024-04-30 17:02:32 on dist-test-slave-x21r"
I20240430 17:02:32.410687 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240430 17:02:32.414070 1047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:32.414924 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240430 17:02:32.415191 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-0-root
uuid: "214687450caa4ed99c39a919d01bb8ab"
format_stamp: "Formatted at 2024-04-30 17:02:32 on dist-test-slave-x21r"
I20240430 17:02:32.415429 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:32.424506 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:32.425361 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:32.426579 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:32.428690 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:32.428858 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:32.429080 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:32.429217 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:32.464157 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38233
I20240430 17:02:32.464241 1109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.65:38233 every 8 connection(s)
I20240430 17:02:32.468376 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:32.477241 1114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:32.478173 1115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:32.479620 1117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:32.481125 413 server_base.cc:1034] running on GCE node
I20240430 17:02:32.481642 1110 heartbeater.cc:347] Connected to a master server at 127.0.103.126:40275
I20240430 17:02:32.481973 1110 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:32.482008 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:32.482357 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:32.482511 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496552482494 us; error 0 us; skew 500 ppm
I20240430 17:02:32.482782 1110 heartbeater.cc:511] Master 127.0.103.126:40275 requested a full tablet report, sending...
I20240430 17:02:32.483069 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:32.485234 983 ts_manager.cc:194] Registered new tserver with Master: 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233)
I20240430 17:02:32.485646 413 webserver.cc:460] Webserver started at http://127.0.103.66:42367/ using document root <none> and password file <none>
I20240430 17:02:32.486101 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:32.486285 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:32.486519 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:32.487048 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54728
I20240430 17:02:32.487596 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-1-root/instance:
uuid: "9a037f7ba58a41be985aa8ea2b9198de"
format_stamp: "Formatted at 2024-04-30 17:02:32 on dist-test-slave-x21r"
I20240430 17:02:32.491677 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240430 17:02:32.494563 1122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:32.495354 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240430 17:02:32.495613 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-1-root
uuid: "9a037f7ba58a41be985aa8ea2b9198de"
format_stamp: "Formatted at 2024-04-30 17:02:32 on dist-test-slave-x21r"
I20240430 17:02:32.495869 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:32.509461 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:32.510417 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:32.511713 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:32.513751 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:32.513948 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:32.514158 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:32.514304 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:32.549852 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:43317
I20240430 17:02:32.549935 1184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.66:43317 every 8 connection(s)
I20240430 17:02:32.554334 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:32.561555 1188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:32.562947 1189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:32.566754 1191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:32.566908 1185 heartbeater.cc:347] Connected to a master server at 127.0.103.126:40275
I20240430 17:02:32.567322 1185 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:32.567615 413 server_base.cc:1034] running on GCE node
I20240430 17:02:32.568241 1185 heartbeater.cc:511] Master 127.0.103.126:40275 requested a full tablet report, sending...
I20240430 17:02:32.568568 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:32.568827 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:32.569044 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496552569027 us; error 0 us; skew 500 ppm
I20240430 17:02:32.569732 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:32.570312 983 ts_manager.cc:194] Registered new tserver with Master: 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317)
I20240430 17:02:32.571812 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54736
I20240430 17:02:32.572381 413 webserver.cc:460] Webserver started at http://127.0.103.67:35261/ using document root <none> and password file <none>
I20240430 17:02:32.573064 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:32.573309 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:32.573637 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:32.574751 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-2-root/instance:
uuid: "98c742cba1854d489581f7f53ebc35a2"
format_stamp: "Formatted at 2024-04-30 17:02:32 on dist-test-slave-x21r"
I20240430 17:02:32.578933 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240430 17:02:32.581769 1196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:32.582458 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240430 17:02:32.582697 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-2-root
uuid: "98c742cba1854d489581f7f53ebc35a2"
format_stamp: "Formatted at 2024-04-30 17:02:32 on dist-test-slave-x21r"
I20240430 17:02:32.582937 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:32.601676 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:32.602645 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:32.603952 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:32.606078 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:32.606290 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:32.606500 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:32.606662 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:32.641870 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:34779
I20240430 17:02:32.641940 1258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.67:34779 every 8 connection(s)
I20240430 17:02:32.654685 1259 heartbeater.cc:347] Connected to a master server at 127.0.103.126:40275
I20240430 17:02:32.655045 1259 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:32.655762 1259 heartbeater.cc:511] Master 127.0.103.126:40275 requested a full tablet report, sending...
I20240430 17:02:32.657452 983 ts_manager.cc:194] Registered new tserver with Master: 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:32.658121 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013107192s
I20240430 17:02:32.658668 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54748
I20240430 17:02:33.489269 1110 heartbeater.cc:503] Master 127.0.103.126:40275 was elected leader, sending a full tablet report...
I20240430 17:02:33.574348 1185 heartbeater.cc:503] Master 127.0.103.126:40275 was elected leader, sending a full tablet report...
I20240430 17:02:33.661093 1259 heartbeater.cc:503] Master 127.0.103.126:40275 was elected leader, sending a full tablet report...
I20240430 17:02:33.689620 413 test_util.cc:268] Using random seed: 1443701243
I20240430 17:02:33.710355 983 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54750:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 17:02:33.712639 983 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 17:02:33.792621 1075 tablet_service.cc:1467] Processing CreateTablet for tablet d331ac9b637247048e3915e06aaf2a25 (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240430 17:02:33.794065 1075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d331ac9b637247048e3915e06aaf2a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.794588 1072 tablet_service.cc:1467] Processing CreateTablet for tablet 69ac9c523f3744d7928c0d14a0625eef (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240430 17:02:33.795912 1072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ac9c523f3744d7928c0d14a0625eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.796103 1073 tablet_service.cc:1467] Processing CreateTablet for tablet 19ae7aeb77f14de988eb80a85aa7ea45 (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240430 17:02:33.797338 1073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ae7aeb77f14de988eb80a85aa7ea45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.797920 1071 tablet_service.cc:1467] Processing CreateTablet for tablet 520fbc131a4b43b7bb23233e1059c51a (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240430 17:02:33.799194 1071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520fbc131a4b43b7bb23233e1059c51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.803810 1074 tablet_service.cc:1467] Processing CreateTablet for tablet 743ddfaf92da4b2896d8146ffe5560ae (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240430 17:02:33.804296 1070 tablet_service.cc:1467] Processing CreateTablet for tablet d1fd9f27d6394358bee83bfd16b7db68 (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240430 17:02:33.805150 1074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743ddfaf92da4b2896d8146ffe5560ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.806640 1069 tablet_service.cc:1467] Processing CreateTablet for tablet 9f89981d63b9449a9722b41db39364bd (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240430 17:02:33.812258 1069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f89981d63b9449a9722b41db39364bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.815455 1070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1fd9f27d6394358bee83bfd16b7db68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.824811 1219 tablet_service.cc:1467] Processing CreateTablet for tablet d1fd9f27d6394358bee83bfd16b7db68 (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240430 17:02:33.824843 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 743ddfaf92da4b2896d8146ffe5560ae (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240430 17:02:33.831213 1219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1fd9f27d6394358bee83bfd16b7db68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.835494 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 520fbc131a4b43b7bb23233e1059c51a (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240430 17:02:33.837432 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520fbc131a4b43b7bb23233e1059c51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.839576 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743ddfaf92da4b2896d8146ffe5560ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.841717 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 19ae7aeb77f14de988eb80a85aa7ea45 (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240430 17:02:33.844772 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 69ac9c523f3744d7928c0d14a0625eef (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240430 17:02:33.846554 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ac9c523f3744d7928c0d14a0625eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.850330 1224 tablet_service.cc:1467] Processing CreateTablet for tablet d331ac9b637247048e3915e06aaf2a25 (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240430 17:02:33.854537 1218 tablet_service.cc:1467] Processing CreateTablet for tablet 9f89981d63b9449a9722b41db39364bd (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240430 17:02:33.866428 1218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f89981d63b9449a9722b41db39364bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.856909 1217 tablet_service.cc:1467] Processing CreateTablet for tablet 640402773e2947058566ccd9f794166c (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240430 17:02:33.872807 1217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640402773e2947058566ccd9f794166c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.879710 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ae7aeb77f14de988eb80a85aa7ea45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.891932 1224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d331ac9b637247048e3915e06aaf2a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:33.897172 1280 tablet_bootstrap.cc:492] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2: Bootstrap starting.
I20240430 17:02:33.910230 1280 tablet_bootstrap.cc:654] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:33.920159 1280 tablet_bootstrap.cc:492] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2: No bootstrap required, opened a new log
I20240430 17:02:33.922854 1280 ts_tablet_manager.cc:1367] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20240430 17:02:33.924556 1279 tablet_bootstrap.cc:492] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab: Bootstrap starting.
I20240430 17:02:33.944802 1280 raft_consensus.cc:348] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:33.945649 1280 raft_consensus.cc:374] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:33.945935 1280 raft_consensus.cc:729] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Initialized, Role: FOLLOWER
I20240430 17:02:33.948254 1279 tablet_bootstrap.cc:654] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:33.946722 1280 consensus_queue.cc:260] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:33.969753 1280 ts_tablet_manager.cc:1398] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2: Time spent starting tablet: real 0.027s user 0.011s sys 0.006s
I20240430 17:02:33.970921 1280 tablet_bootstrap.cc:492] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2: Bootstrap starting.
I20240430 17:02:33.972220 1279 tablet_bootstrap.cc:492] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab: No bootstrap required, opened a new log
I20240430 17:02:33.972707 1279 ts_tablet_manager.cc:1367] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.020s
I20240430 17:02:33.975363 1279 raft_consensus.cc:348] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:33.976037 1280 tablet_bootstrap.cc:654] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:33.976158 1279 raft_consensus.cc:374] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:33.976536 1279 raft_consensus.cc:729] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214687450caa4ed99c39a919d01bb8ab, State: Initialized, Role: FOLLOWER
I20240430 17:02:33.977371 1279 consensus_queue.cc:260] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:33.982028 1280 tablet_bootstrap.cc:492] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2: No bootstrap required, opened a new log
I20240430 17:02:33.982523 1280 ts_tablet_manager.cc:1367] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240430 17:02:33.985147 1280 raft_consensus.cc:348] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:33.985800 1280 raft_consensus.cc:374] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:33.986116 1280 raft_consensus.cc:729] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Initialized, Role: FOLLOWER
I20240430 17:02:33.986799 1280 consensus_queue.cc:260] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:33.988942 1280 ts_tablet_manager.cc:1398] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240430 17:02:33.989806 1280 tablet_bootstrap.cc:492] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2: Bootstrap starting.
I20240430 17:02:33.995532 1280 tablet_bootstrap.cc:654] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.000883 1280 tablet_bootstrap.cc:492] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2: No bootstrap required, opened a new log
I20240430 17:02:34.001422 1280 ts_tablet_manager.cc:1367] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240430 17:02:34.003713 1280 raft_consensus.cc:348] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.004422 1280 raft_consensus.cc:374] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.004675 1280 raft_consensus.cc:729] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.005398 1280 consensus_queue.cc:260] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.007489 1280 ts_tablet_manager.cc:1398] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240430 17:02:34.008391 1280 tablet_bootstrap.cc:492] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2: Bootstrap starting.
I20240430 17:02:34.013478 1280 tablet_bootstrap.cc:654] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.017926 1148 tablet_service.cc:1467] Processing CreateTablet for tablet 19ae7aeb77f14de988eb80a85aa7ea45 (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240430 17:02:34.019392 1148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ae7aeb77f14de988eb80a85aa7ea45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:34.019326 1150 tablet_service.cc:1467] Processing CreateTablet for tablet d331ac9b637247048e3915e06aaf2a25 (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240430 17:02:34.020675 1150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d331ac9b637247048e3915e06aaf2a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:34.024349 1149 tablet_service.cc:1467] Processing CreateTablet for tablet 743ddfaf92da4b2896d8146ffe5560ae (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240430 17:02:34.024767 1145 tablet_service.cc:1467] Processing CreateTablet for tablet d1fd9f27d6394358bee83bfd16b7db68 (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240430 17:02:34.025842 1149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743ddfaf92da4b2896d8146ffe5560ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:34.026520 1145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1fd9f27d6394358bee83bfd16b7db68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:34.026456 1146 tablet_service.cc:1467] Processing CreateTablet for tablet 520fbc131a4b43b7bb23233e1059c51a (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240430 17:02:34.026636 1143 tablet_service.cc:1467] Processing CreateTablet for tablet 640402773e2947058566ccd9f794166c (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240430 17:02:34.028378 1146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520fbc131a4b43b7bb23233e1059c51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:34.031430 1143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640402773e2947058566ccd9f794166c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:34.048652 1280 tablet_bootstrap.cc:492] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2: No bootstrap required, opened a new log
I20240430 17:02:34.049557 1280 ts_tablet_manager.cc:1367] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.000s
I20240430 17:02:34.052632 1280 raft_consensus.cc:348] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } }
I20240430 17:02:34.053467 1280 raft_consensus.cc:374] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.053804 1280 raft_consensus.cc:729] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.057650 1283 raft_consensus.cc:482] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.058090 1280 consensus_queue.cc:260] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } }
I20240430 17:02:34.058105 1283 raft_consensus.cc:504] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.060320 1288 tablet_bootstrap.cc:492] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de: Bootstrap starting.
I20240430 17:02:34.072302 1283 leader_election.cc:288] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317)
I20240430 17:02:34.099933 1280 ts_tablet_manager.cc:1398] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2: Time spent starting tablet: real 0.050s user 0.014s sys 0.015s
I20240430 17:02:34.101253 1280 tablet_bootstrap.cc:492] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2: Bootstrap starting.
I20240430 17:02:34.102063 1073 tablet_service.cc:1467] Processing CreateTablet for tablet 640402773e2947058566ccd9f794166c (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240430 17:02:34.103343 1073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640402773e2947058566ccd9f794166c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:34.106773 1280 tablet_bootstrap.cc:654] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.114936 1279 ts_tablet_manager.cc:1398] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab: Time spent starting tablet: real 0.142s user 0.007s sys 0.003s
I20240430 17:02:34.132018 1279 tablet_bootstrap.cc:492] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab: Bootstrap starting.
I20240430 17:02:34.119642 1288 tablet_bootstrap.cc:654] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.121891 1144 tablet_service.cc:1467] Processing CreateTablet for tablet 9f89981d63b9449a9722b41db39364bd (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240430 17:02:34.134408 1144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f89981d63b9449a9722b41db39364bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:34.136952 1147 tablet_service.cc:1467] Processing CreateTablet for tablet 69ac9c523f3744d7928c0d14a0625eef (DEFAULT_TABLE table=test-workload [id=adfc1859fc4c4c4daa41619745129a06]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240430 17:02:34.138859 1147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ac9c523f3744d7928c0d14a0625eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:34.139887 1289 raft_consensus.cc:482] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.140333 1289 raft_consensus.cc:504] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
W20240430 17:02:34.147387 1200 leader_election.cc:341] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233): Not found: Tablet not found: 640402773e2947058566ccd9f794166c
I20240430 17:02:34.149164 1085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fd9f27d6394358bee83bfd16b7db68" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab" is_pre_election: true
W20240430 17:02:34.150652 1200 leader_election.cc:341] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233): Illegal state: must be running to vote when last-logged opid is not known
I20240430 17:02:34.150794 1279 tablet_bootstrap.cc:654] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.159286 1289 leader_election.cc:288] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317)
I20240430 17:02:34.168439 1280 tablet_bootstrap.cc:492] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2: No bootstrap required, opened a new log
I20240430 17:02:34.168931 1280 ts_tablet_manager.cc:1367] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2: Time spent bootstrapping tablet: real 0.068s user 0.024s sys 0.004s
I20240430 17:02:34.170639 1280 raft_consensus.cc:348] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.174799 1280 raft_consensus.cc:374] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.175091 1280 raft_consensus.cc:729] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.175383 1160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640402773e2947058566ccd9f794166c" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de" is_pre_election: true
I20240430 17:02:34.175453 1159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fd9f27d6394358bee83bfd16b7db68" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de" is_pre_election: true
I20240430 17:02:34.175829 1280 consensus_queue.cc:260] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
W20240430 17:02:34.177455 1199 leader_election.cc:341] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317): Illegal state: must be running to vote when last-logged opid is not known
I20240430 17:02:34.177891 1199 leader_election.cc:302] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98c742cba1854d489581f7f53ebc35a2; no voters: 214687450caa4ed99c39a919d01bb8ab, 9a037f7ba58a41be985aa8ea2b9198de
W20240430 17:02:34.178982 1199 leader_election.cc:341] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317): Illegal state: must be running to vote when last-logged opid is not known
I20240430 17:02:34.179405 1199 leader_election.cc:302] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98c742cba1854d489581f7f53ebc35a2; no voters: 214687450caa4ed99c39a919d01bb8ab, 9a037f7ba58a41be985aa8ea2b9198de
I20240430 17:02:34.180706 1283 raft_consensus.cc:2726] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 17:02:34.181267 1283 raft_consensus.cc:2726] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 17:02:34.185325 1280 ts_tablet_manager.cc:1398] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240430 17:02:34.186282 1280 tablet_bootstrap.cc:492] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2: Bootstrap starting.
I20240430 17:02:34.192396 1288 tablet_bootstrap.cc:492] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de: No bootstrap required, opened a new log
I20240430 17:02:34.192873 1288 ts_tablet_manager.cc:1367] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de: Time spent bootstrapping tablet: real 0.133s user 0.020s sys 0.016s
I20240430 17:02:34.193238 1279 tablet_bootstrap.cc:492] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab: No bootstrap required, opened a new log
I20240430 17:02:34.193414 1280 tablet_bootstrap.cc:654] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.193692 1279 ts_tablet_manager.cc:1367] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.008s
I20240430 17:02:34.195233 1288 raft_consensus.cc:348] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.196014 1288 raft_consensus.cc:374] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.196362 1288 raft_consensus.cc:729] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a037f7ba58a41be985aa8ea2b9198de, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.196108 1279 raft_consensus.cc:348] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } }
I20240430 17:02:34.196957 1279 raft_consensus.cc:374] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.197254 1279 raft_consensus.cc:729] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214687450caa4ed99c39a919d01bb8ab, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.197187 1288 consensus_queue.cc:260] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.197854 1280 tablet_bootstrap.cc:492] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2: No bootstrap required, opened a new log
I20240430 17:02:34.198274 1280 ts_tablet_manager.cc:1367] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20240430 17:02:34.198113 1279 consensus_queue.cc:260] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } }
I20240430 17:02:34.200279 1288 ts_tablet_manager.cc:1398] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240430 17:02:34.201175 1288 tablet_bootstrap.cc:492] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de: Bootstrap starting.
I20240430 17:02:34.201239 1280 raft_consensus.cc:348] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.202046 1280 raft_consensus.cc:374] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.202330 1280 raft_consensus.cc:729] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.203207 1280 consensus_queue.cc:260] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.205914 1280 ts_tablet_manager.cc:1398] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20240430 17:02:34.206873 1280 tablet_bootstrap.cc:492] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2: Bootstrap starting.
I20240430 17:02:34.208484 1288 tablet_bootstrap.cc:654] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.212477 1280 tablet_bootstrap.cc:654] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.213965 1279 ts_tablet_manager.cc:1398] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20240430 17:02:34.224560 1279 tablet_bootstrap.cc:492] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab: Bootstrap starting.
I20240430 17:02:34.224475 1289 raft_consensus.cc:482] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.225369 1289 raft_consensus.cc:504] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.228960 1085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f89981d63b9449a9722b41db39364bd" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab" is_pre_election: true
W20240430 17:02:34.230890 1200 leader_election.cc:341] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233): Illegal state: must be running to vote when last-logged opid is not known
I20240430 17:02:34.232808 1289 leader_election.cc:288] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317)
I20240430 17:02:34.234222 1159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f89981d63b9449a9722b41db39364bd" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de" is_pre_election: true
I20240430 17:02:34.234634 1279 tablet_bootstrap.cc:654] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab: Neither blocks nor log segments found. Creating new log.
W20240430 17:02:34.235318 1199 leader_election.cc:341] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317): Illegal state: must be running to vote when last-logged opid is not known
I20240430 17:02:34.235909 1288 tablet_bootstrap.cc:492] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de: No bootstrap required, opened a new log
I20240430 17:02:34.235749 1199 leader_election.cc:302] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98c742cba1854d489581f7f53ebc35a2; no voters: 214687450caa4ed99c39a919d01bb8ab, 9a037f7ba58a41be985aa8ea2b9198de
I20240430 17:02:34.236460 1288 ts_tablet_manager.cc:1367] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de: Time spent bootstrapping tablet: real 0.035s user 0.020s sys 0.013s
I20240430 17:02:34.236858 1289 raft_consensus.cc:2726] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 17:02:34.239475 1288 raft_consensus.cc:348] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.240247 1288 raft_consensus.cc:374] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.240464 1288 raft_consensus.cc:729] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a037f7ba58a41be985aa8ea2b9198de, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.240898 1280 tablet_bootstrap.cc:492] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2: No bootstrap required, opened a new log
I20240430 17:02:34.241014 1288 consensus_queue.cc:260] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.241446 1279 tablet_bootstrap.cc:492] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab: No bootstrap required, opened a new log
I20240430 17:02:34.241410 1280 ts_tablet_manager.cc:1367] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.016s
I20240430 17:02:34.242014 1279 ts_tablet_manager.cc:1367] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.000s
I20240430 17:02:34.243376 1288 ts_tablet_manager.cc:1398] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240430 17:02:34.244442 1288 tablet_bootstrap.cc:492] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de: Bootstrap starting.
I20240430 17:02:34.244537 1280 raft_consensus.cc:348] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.245324 1280 raft_consensus.cc:374] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.245606 1280 raft_consensus.cc:729] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.246356 1280 consensus_queue.cc:260] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.248553 1280 ts_tablet_manager.cc:1398] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240430 17:02:34.249405 1280 tablet_bootstrap.cc:492] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2: Bootstrap starting.
I20240430 17:02:34.249559 1288 tablet_bootstrap.cc:654] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.255939 1280 tablet_bootstrap.cc:654] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.254947 1279 raft_consensus.cc:348] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.256310 1279 raft_consensus.cc:374] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.256594 1279 raft_consensus.cc:729] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214687450caa4ed99c39a919d01bb8ab, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.257922 1279 consensus_queue.cc:260] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.260888 1289 raft_consensus.cc:482] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.261356 1289 raft_consensus.cc:504] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.264168 1085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ae7aeb77f14de988eb80a85aa7ea45" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab" is_pre_election: true
I20240430 17:02:34.264858 1085 raft_consensus.cc:2454] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98c742cba1854d489581f7f53ebc35a2 in term 0.
I20240430 17:02:34.265925 1200 leader_election.cc:302] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 98c742cba1854d489581f7f53ebc35a2; no voters:
I20240430 17:02:34.267957 1283 raft_consensus.cc:2781] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:02:34.268325 1283 raft_consensus.cc:482] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:34.268640 1283 raft_consensus.cc:3037] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.270249 1279 ts_tablet_manager.cc:1398] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20240430 17:02:34.271781 1289 leader_election.cc:288] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317)
I20240430 17:02:34.272248 1289 raft_consensus.cc:482] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.272652 1289 raft_consensus.cc:504] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.274508 1289 leader_election.cc:288] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317)
I20240430 17:02:34.276137 1159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ae7aeb77f14de988eb80a85aa7ea45" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de" is_pre_election: true
I20240430 17:02:34.276409 1160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743ddfaf92da4b2896d8146ffe5560ae" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de" is_pre_election: true
W20240430 17:02:34.277925 1199 leader_election.cc:341] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317): Illegal state: must be running to vote when last-logged opid is not known
W20240430 17:02:34.278617 1199 leader_election.cc:341] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317): Illegal state: must be running to vote when last-logged opid is not known
I20240430 17:02:34.279968 1279 tablet_bootstrap.cc:492] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab: Bootstrap starting.
I20240430 17:02:34.281124 1283 raft_consensus.cc:504] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.284081 1159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ae7aeb77f14de988eb80a85aa7ea45" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de"
I20240430 17:02:34.285193 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ae7aeb77f14de988eb80a85aa7ea45" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab"
I20240430 17:02:34.286253 1279 tablet_bootstrap.cc:654] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.286329 1084 raft_consensus.cc:3037] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Advancing to term 1
W20240430 17:02:34.288040 1199 leader_election.cc:341] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317): Illegal state: must be running to vote when last-logged opid is not known
I20240430 17:02:34.288859 1085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743ddfaf92da4b2896d8146ffe5560ae" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab" is_pre_election: true
W20240430 17:02:34.290364 1200 leader_election.cc:341] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233): Illegal state: must be running to vote when last-logged opid is not known
I20240430 17:02:34.290832 1200 leader_election.cc:302] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98c742cba1854d489581f7f53ebc35a2; no voters: 214687450caa4ed99c39a919d01bb8ab, 9a037f7ba58a41be985aa8ea2b9198de
I20240430 17:02:34.292186 1280 tablet_bootstrap.cc:492] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2: No bootstrap required, opened a new log
I20240430 17:02:34.292691 1280 ts_tablet_manager.cc:1367] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.000s
I20240430 17:02:34.293208 1084 raft_consensus.cc:2454] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98c742cba1854d489581f7f53ebc35a2 in term 1.
I20240430 17:02:34.295083 1200 leader_election.cc:302] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 98c742cba1854d489581f7f53ebc35a2; no voters: 9a037f7ba58a41be985aa8ea2b9198de
I20240430 17:02:34.295964 1289 raft_consensus.cc:2781] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:34.296218 1288 tablet_bootstrap.cc:492] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de: No bootstrap required, opened a new log
I20240430 17:02:34.296804 1288 ts_tablet_manager.cc:1367] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de: Time spent bootstrapping tablet: real 0.053s user 0.022s sys 0.019s
I20240430 17:02:34.293535 1283 leader_election.cc:288] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 election: Requested vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317)
I20240430 17:02:34.298944 1280 raft_consensus.cc:348] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.299916 1280 raft_consensus.cc:374] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.300252 1280 raft_consensus.cc:729] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.300457 1288 raft_consensus.cc:348] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.301432 1288 raft_consensus.cc:374] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.301205 1280 consensus_queue.cc:260] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.301918 1288 raft_consensus.cc:729] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a037f7ba58a41be985aa8ea2b9198de, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.303268 1288 consensus_queue.cc:260] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.304325 1280 ts_tablet_manager.cc:1398] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20240430 17:02:34.294489 1302 raft_consensus.cc:2726] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 17:02:34.312939 1288 ts_tablet_manager.cc:1398] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240430 17:02:34.313938 1288 tablet_bootstrap.cc:492] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de: Bootstrap starting.
W20240430 17:02:34.317533 1031 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240430 17:02:34.327453 1296 raft_consensus.cc:482] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.328256 1288 tablet_bootstrap.cc:654] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.328037 1296 raft_consensus.cc:504] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.320796 1289 raft_consensus.cc:686] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Becoming Leader. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Running, Role: LEADER
I20240430 17:02:34.331293 1036 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:02:34.332383 1036 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:02:34.331885 1289 consensus_queue.cc:237] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.331288 1279 tablet_bootstrap.cc:492] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab: No bootstrap required, opened a new log
I20240430 17:02:34.335988 1279 ts_tablet_manager.cc:1367] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.018s
I20240430 17:02:34.337311 1296 leader_election.cc:288] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:34.338783 1279 raft_consensus.cc:348] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.340018 1279 raft_consensus.cc:374] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.340451 1279 raft_consensus.cc:729] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214687450caa4ed99c39a919d01bb8ab, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.341360 1279 consensus_queue.cc:260] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.344115 1279 ts_tablet_manager.cc:1398] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20240430 17:02:34.345100 1279 tablet_bootstrap.cc:492] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab: Bootstrap starting.
I20240430 17:02:34.351161 1279 tablet_bootstrap.cc:654] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.357317 1288 tablet_bootstrap.cc:492] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de: No bootstrap required, opened a new log
I20240430 17:02:34.358011 1288 ts_tablet_manager.cc:1367] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.021s
I20240430 17:02:34.363014 1288 raft_consensus.cc:348] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.364048 1288 raft_consensus.cc:374] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.364418 1288 raft_consensus.cc:729] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a037f7ba58a41be985aa8ea2b9198de, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.365437 1288 consensus_queue.cc:260] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.369143 1279 tablet_bootstrap.cc:492] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab: No bootstrap required, opened a new log
I20240430 17:02:34.369714 1279 ts_tablet_manager.cc:1367] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.009s
I20240430 17:02:34.375607 1288 ts_tablet_manager.cc:1398] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de: Time spent starting tablet: real 0.017s user 0.008s sys 0.001s
I20240430 17:02:34.375385 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ae7aeb77f14de988eb80a85aa7ea45" candidate_uuid: "9a037f7ba58a41be985aa8ea2b9198de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab" is_pre_election: true
I20240430 17:02:34.377135 1084 raft_consensus.cc:2379] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a037f7ba58a41be985aa8ea2b9198de in current term 1: Already voted for candidate 98c742cba1854d489581f7f53ebc35a2 in this term.
I20240430 17:02:34.377507 1288 tablet_bootstrap.cc:492] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de: Bootstrap starting.
I20240430 17:02:34.377045 982 catalog_manager.cc:5484] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "98c742cba1854d489581f7f53ebc35a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } health_report { overall_health: HEALTHY } } }
I20240430 17:02:34.373823 1279 raft_consensus.cc:348] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.380515 1279 raft_consensus.cc:374] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.380856 1279 raft_consensus.cc:729] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214687450caa4ed99c39a919d01bb8ab, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.380710 1234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ae7aeb77f14de988eb80a85aa7ea45" candidate_uuid: "9a037f7ba58a41be985aa8ea2b9198de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c742cba1854d489581f7f53ebc35a2" is_pre_election: true
I20240430 17:02:34.381546 1279 consensus_queue.cc:260] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.382644 1125 leader_election.cc:302] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a037f7ba58a41be985aa8ea2b9198de; no voters: 214687450caa4ed99c39a919d01bb8ab, 98c742cba1854d489581f7f53ebc35a2
I20240430 17:02:34.383785 1279 ts_tablet_manager.cc:1398] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240430 17:02:34.384303 1288 tablet_bootstrap.cc:654] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.384781 1279 tablet_bootstrap.cc:492] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab: Bootstrap starting.
I20240430 17:02:34.391736 1288 tablet_bootstrap.cc:492] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de: No bootstrap required, opened a new log
I20240430 17:02:34.392261 1288 ts_tablet_manager.cc:1367] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20240430 17:02:34.393208 1296 raft_consensus.cc:3037] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.393836 1279 tablet_bootstrap.cc:654] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.395404 1288 raft_consensus.cc:348] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.396167 1288 raft_consensus.cc:374] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.396483 1288 raft_consensus.cc:729] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a037f7ba58a41be985aa8ea2b9198de, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.397151 1288 consensus_queue.cc:260] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.401939 1296 raft_consensus.cc:2726] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 17:02:34.411201 1279 tablet_bootstrap.cc:492] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab: No bootstrap required, opened a new log
I20240430 17:02:34.411638 1288 ts_tablet_manager.cc:1398] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de: Time spent starting tablet: real 0.019s user 0.009s sys 0.010s
I20240430 17:02:34.411689 1279 ts_tablet_manager.cc:1367] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab: Time spent bootstrapping tablet: real 0.027s user 0.020s sys 0.002s
I20240430 17:02:34.412812 1288 tablet_bootstrap.cc:492] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de: Bootstrap starting.
I20240430 17:02:34.414610 1279 raft_consensus.cc:348] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.415343 1279 raft_consensus.cc:374] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.415580 1279 raft_consensus.cc:729] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214687450caa4ed99c39a919d01bb8ab, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.416170 1279 consensus_queue.cc:260] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.418084 1279 ts_tablet_manager.cc:1398] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240430 17:02:34.418849 1279 tablet_bootstrap.cc:492] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab: Bootstrap starting.
I20240430 17:02:34.419392 1288 tablet_bootstrap.cc:654] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.419260 1289 raft_consensus.cc:482] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.420662 1289 raft_consensus.cc:504] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.423460 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520fbc131a4b43b7bb23233e1059c51a" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab" is_pre_election: true
I20240430 17:02:34.424777 1279 tablet_bootstrap.cc:654] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab: Neither blocks nor log segments found. Creating new log.
W20240430 17:02:34.424841 1200 leader_election.cc:341] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233): Illegal state: must be running to vote when last-logged opid is not known
I20240430 17:02:34.427238 1289 leader_election.cc:288] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317)
I20240430 17:02:34.428671 1159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520fbc131a4b43b7bb23233e1059c51a" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de" is_pre_election: true
I20240430 17:02:34.430243 1159 raft_consensus.cc:2454] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98c742cba1854d489581f7f53ebc35a2 in term 0.
I20240430 17:02:34.431696 1199 leader_election.cc:302] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98c742cba1854d489581f7f53ebc35a2, 9a037f7ba58a41be985aa8ea2b9198de; no voters: 214687450caa4ed99c39a919d01bb8ab
I20240430 17:02:34.434616 1304 raft_consensus.cc:2781] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:02:34.433908 1289 raft_consensus.cc:482] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.435602 1304 raft_consensus.cc:482] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:34.437264 1304 raft_consensus.cc:3037] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.437455 1289 raft_consensus.cc:504] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } }
I20240430 17:02:34.440759 1159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d331ac9b637247048e3915e06aaf2a25" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de" is_pre_election: true
I20240430 17:02:34.441412 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d331ac9b637247048e3915e06aaf2a25" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab" is_pre_election: true
W20240430 17:02:34.441908 1199 leader_election.cc:341] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317): Illegal state: must be running to vote when last-logged opid is not known
I20240430 17:02:34.442288 1084 raft_consensus.cc:2454] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98c742cba1854d489581f7f53ebc35a2 in term 0.
I20240430 17:02:34.443512 1200 leader_election.cc:302] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 98c742cba1854d489581f7f53ebc35a2; no voters: 9a037f7ba58a41be985aa8ea2b9198de
I20240430 17:02:34.437252 1279 tablet_bootstrap.cc:492] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab: No bootstrap required, opened a new log
I20240430 17:02:34.443686 1304 raft_consensus.cc:504] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.444562 1279 ts_tablet_manager.cc:1367] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.008s
I20240430 17:02:34.444553 1302 raft_consensus.cc:2781] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:02:34.445063 1302 raft_consensus.cc:482] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:34.445322 1302 raft_consensus.cc:3037] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.447455 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520fbc131a4b43b7bb23233e1059c51a" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab"
I20240430 17:02:34.448017 1159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520fbc131a4b43b7bb23233e1059c51a" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de"
W20240430 17:02:34.448781 1200 leader_election.cc:341] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233): Illegal state: must be running to vote when last-logged opid is not known
I20240430 17:02:34.448781 1159 raft_consensus.cc:3037] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.449671 1289 leader_election.cc:288] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317), 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233)
I20240430 17:02:34.450398 1304 leader_election.cc:288] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 election: Requested vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317)
I20240430 17:02:34.451148 1288 tablet_bootstrap.cc:492] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de: No bootstrap required, opened a new log
I20240430 17:02:34.451576 1288 ts_tablet_manager.cc:1367] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de: Time spent bootstrapping tablet: real 0.039s user 0.025s sys 0.011s
I20240430 17:02:34.451756 1302 raft_consensus.cc:504] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } }
I20240430 17:02:34.453656 1279 raft_consensus.cc:348] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.454398 1279 raft_consensus.cc:374] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.454092 1288 raft_consensus.cc:348] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } }
I20240430 17:02:34.454402 1160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d331ac9b637247048e3915e06aaf2a25" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de"
I20240430 17:02:34.455070 1288 raft_consensus.cc:374] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.454892 1279 raft_consensus.cc:729] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214687450caa4ed99c39a919d01bb8ab, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.455492 1288 raft_consensus.cc:729] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a037f7ba58a41be985aa8ea2b9198de, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.456285 1159 raft_consensus.cc:2454] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98c742cba1854d489581f7f53ebc35a2 in term 1.
I20240430 17:02:34.456190 1279 consensus_queue.cc:260] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.456380 1288 consensus_queue.cc:260] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } }
I20240430 17:02:34.457098 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d331ac9b637247048e3915e06aaf2a25" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab"
I20240430 17:02:34.457670 1084 raft_consensus.cc:3037] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.458458 1288 ts_tablet_manager.cc:1398] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240430 17:02:34.458685 1160 raft_consensus.cc:3037] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.458504 1279 ts_tablet_manager.cc:1398] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20240430 17:02:34.460390 1279 tablet_bootstrap.cc:492] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab: Bootstrap starting.
I20240430 17:02:34.460860 1288 tablet_bootstrap.cc:492] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de: Bootstrap starting.
I20240430 17:02:34.462633 1302 leader_election.cc:288] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 election: Requested vote from peers 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317), 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233)
I20240430 17:02:34.458534 1199 leader_election.cc:302] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98c742cba1854d489581f7f53ebc35a2, 9a037f7ba58a41be985aa8ea2b9198de; no voters: 214687450caa4ed99c39a919d01bb8ab
I20240430 17:02:34.463973 1302 raft_consensus.cc:2781] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:34.465188 1302 raft_consensus.cc:686] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Becoming Leader. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Running, Role: LEADER
I20240430 17:02:34.465909 1302 consensus_queue.cc:237] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.466470 1160 raft_consensus.cc:2454] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98c742cba1854d489581f7f53ebc35a2 in term 1.
I20240430 17:02:34.471313 1084 raft_consensus.cc:2454] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98c742cba1854d489581f7f53ebc35a2 in term 1.
I20240430 17:02:34.473502 1199 leader_election.cc:302] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98c742cba1854d489581f7f53ebc35a2, 9a037f7ba58a41be985aa8ea2b9198de; no voters:
I20240430 17:02:34.474598 1304 raft_consensus.cc:2781] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:34.475148 1304 raft_consensus.cc:686] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Becoming Leader. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Running, Role: LEADER
I20240430 17:02:34.475757 1279 tablet_bootstrap.cc:654] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.475880 1304 consensus_queue.cc:237] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } }
I20240430 17:02:34.484844 1288 tablet_bootstrap.cc:654] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.486186 982 catalog_manager.cc:5484] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "98c742cba1854d489581f7f53ebc35a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } health_report { overall_health: HEALTHY } } }
I20240430 17:02:34.489701 1279 tablet_bootstrap.cc:492] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab: No bootstrap required, opened a new log
I20240430 17:02:34.489948 1288 tablet_bootstrap.cc:492] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de: No bootstrap required, opened a new log
I20240430 17:02:34.490182 1279 ts_tablet_manager.cc:1367] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.006s
I20240430 17:02:34.490371 1288 ts_tablet_manager.cc:1367] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.005s
I20240430 17:02:34.493063 1279 raft_consensus.cc:348] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.493310 1288 raft_consensus.cc:348] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.493769 1279 raft_consensus.cc:374] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.493978 1288 raft_consensus.cc:374] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.494062 1279 raft_consensus.cc:729] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214687450caa4ed99c39a919d01bb8ab, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.494349 1288 raft_consensus.cc:729] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a037f7ba58a41be985aa8ea2b9198de, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.494720 1279 consensus_queue.cc:260] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.495733 1288 consensus_queue.cc:260] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.497001 1279 ts_tablet_manager.cc:1398] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20240430 17:02:34.498605 1288 ts_tablet_manager.cc:1398] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20240430 17:02:34.499840 1288 tablet_bootstrap.cc:492] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de: Bootstrap starting.
I20240430 17:02:34.501631 1315 raft_consensus.cc:482] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.501572 980 catalog_manager.cc:5484] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "98c742cba1854d489581f7f53ebc35a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } health_report { overall_health: UNKNOWN } } }
I20240430 17:02:34.502135 1315 raft_consensus.cc:504] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.504320 1315 leader_election.cc:288] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:34.506382 1288 tablet_bootstrap.cc:654] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:34.506608 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fd9f27d6394358bee83bfd16b7db68" candidate_uuid: "9a037f7ba58a41be985aa8ea2b9198de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab" is_pre_election: true
I20240430 17:02:34.507261 1084 raft_consensus.cc:2454] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a037f7ba58a41be985aa8ea2b9198de in term 0.
I20240430 17:02:34.509260 1126 leader_election.cc:302] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 9a037f7ba58a41be985aa8ea2b9198de; no voters:
I20240430 17:02:34.510314 1234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fd9f27d6394358bee83bfd16b7db68" candidate_uuid: "9a037f7ba58a41be985aa8ea2b9198de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c742cba1854d489581f7f53ebc35a2" is_pre_election: true
I20240430 17:02:34.510948 1234 raft_consensus.cc:2454] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a037f7ba58a41be985aa8ea2b9198de in term 0.
I20240430 17:02:34.514300 1315 raft_consensus.cc:2781] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:02:34.514801 1315 raft_consensus.cc:482] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:34.515611 1315 raft_consensus.cc:3037] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.518121 1288 tablet_bootstrap.cc:492] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de: No bootstrap required, opened a new log
I20240430 17:02:34.518587 1288 ts_tablet_manager.cc:1367] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20240430 17:02:34.521615 1288 raft_consensus.cc:348] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.522281 1288 raft_consensus.cc:374] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:34.522547 1288 raft_consensus.cc:729] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a037f7ba58a41be985aa8ea2b9198de, State: Initialized, Role: FOLLOWER
I20240430 17:02:34.523289 1315 raft_consensus.cc:504] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.523182 1288 consensus_queue.cc:260] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.525316 1288 ts_tablet_manager.cc:1398] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240430 17:02:34.525851 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fd9f27d6394358bee83bfd16b7db68" candidate_uuid: "9a037f7ba58a41be985aa8ea2b9198de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab"
I20240430 17:02:34.526298 1084 raft_consensus.cc:3037] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.531888 1084 raft_consensus.cc:2454] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a037f7ba58a41be985aa8ea2b9198de in term 1.
I20240430 17:02:34.532877 1126 leader_election.cc:302] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 9a037f7ba58a41be985aa8ea2b9198de; no voters:
I20240430 17:02:34.533241 1234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fd9f27d6394358bee83bfd16b7db68" candidate_uuid: "9a037f7ba58a41be985aa8ea2b9198de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c742cba1854d489581f7f53ebc35a2"
I20240430 17:02:34.532347 1315 leader_election.cc:288] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [CANDIDATE]: Term 1 election: Requested vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:34.533700 1296 raft_consensus.cc:2781] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:34.533823 1234 raft_consensus.cc:3037] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.534338 1296 raft_consensus.cc:686] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 LEADER]: Becoming Leader. State: Replica: 9a037f7ba58a41be985aa8ea2b9198de, State: Running, Role: LEADER
I20240430 17:02:34.535243 1296 consensus_queue.cc:237] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.537238 1285 raft_consensus.cc:482] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.537717 1285 raft_consensus.cc:504] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.542006 1285 leader_election.cc:288] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317), 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:34.540670 1234 raft_consensus.cc:2454] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a037f7ba58a41be985aa8ea2b9198de in term 1.
I20240430 17:02:34.545617 982 catalog_manager.cc:5484] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66). New cstate: current_term: 1 leader_uuid: "9a037f7ba58a41be985aa8ea2b9198de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } health_report { overall_health: UNKNOWN } } }
I20240430 17:02:34.555181 1160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640402773e2947058566ccd9f794166c" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de" is_pre_election: true
I20240430 17:02:34.555852 1160 raft_consensus.cc:2454] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214687450caa4ed99c39a919d01bb8ab in term 0.
I20240430 17:02:34.556058 1234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640402773e2947058566ccd9f794166c" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c742cba1854d489581f7f53ebc35a2" is_pre_election: true
I20240430 17:02:34.556553 1234 raft_consensus.cc:2454] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214687450caa4ed99c39a919d01bb8ab in term 0.
I20240430 17:02:34.556979 1050 leader_election.cc:302] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 9a037f7ba58a41be985aa8ea2b9198de; no voters:
I20240430 17:02:34.557251 1285 raft_consensus.cc:482] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.557687 1285 raft_consensus.cc:504] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.559350 1326 raft_consensus.cc:2781] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:02:34.559676 1326 raft_consensus.cc:482] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:34.559918 1326 raft_consensus.cc:3037] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.560222 1285 leader_election.cc:288] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317), 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:34.561076 1234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743ddfaf92da4b2896d8146ffe5560ae" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c742cba1854d489581f7f53ebc35a2" is_pre_election: true
I20240430 17:02:34.561277 1160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743ddfaf92da4b2896d8146ffe5560ae" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de" is_pre_election: true
I20240430 17:02:34.561530 1234 raft_consensus.cc:2454] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214687450caa4ed99c39a919d01bb8ab in term 0.
I20240430 17:02:34.561928 1160 raft_consensus.cc:2454] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214687450caa4ed99c39a919d01bb8ab in term 0.
I20240430 17:02:34.562299 1050 leader_election.cc:302] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 98c742cba1854d489581f7f53ebc35a2; no voters:
I20240430 17:02:34.562866 1285 raft_consensus.cc:2781] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:02:34.563419 1285 raft_consensus.cc:482] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:34.563664 1285 raft_consensus.cc:3037] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.565972 1326 raft_consensus.cc:504] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.567965 1326 leader_election.cc:288] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 election: Requested vote from peers 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317), 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:34.568609 1285 raft_consensus.cc:504] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.568768 1160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640402773e2947058566ccd9f794166c" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de"
I20240430 17:02:34.569056 1234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640402773e2947058566ccd9f794166c" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c742cba1854d489581f7f53ebc35a2"
I20240430 17:02:34.569411 1160 raft_consensus.cc:3037] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.569598 1234 raft_consensus.cc:3037] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.570261 1285 leader_election.cc:288] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 election: Requested vote from peers 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317), 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:34.570866 1159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743ddfaf92da4b2896d8146ffe5560ae" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de"
I20240430 17:02:34.571329 1159 raft_consensus.cc:3037] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.575332 1233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743ddfaf92da4b2896d8146ffe5560ae" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c742cba1854d489581f7f53ebc35a2"
I20240430 17:02:34.575695 1234 raft_consensus.cc:2454] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214687450caa4ed99c39a919d01bb8ab in term 1.
I20240430 17:02:34.576020 1233 raft_consensus.cc:3037] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.576275 1159 raft_consensus.cc:2454] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214687450caa4ed99c39a919d01bb8ab in term 1.
I20240430 17:02:34.576763 1050 leader_election.cc:302] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 98c742cba1854d489581f7f53ebc35a2; no voters:
I20240430 17:02:34.577363 1160 raft_consensus.cc:2454] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214687450caa4ed99c39a919d01bb8ab in term 1.
I20240430 17:02:34.577569 1285 raft_consensus.cc:2781] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:34.577723 1050 leader_election.cc:302] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 9a037f7ba58a41be985aa8ea2b9198de; no voters:
I20240430 17:02:34.578321 1285 raft_consensus.cc:686] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 1 LEADER]: Becoming Leader. State: Replica: 214687450caa4ed99c39a919d01bb8ab, State: Running, Role: LEADER
I20240430 17:02:34.579181 1326 raft_consensus.cc:2781] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:34.579712 1326 raft_consensus.cc:686] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 1 LEADER]: Becoming Leader. State: Replica: 214687450caa4ed99c39a919d01bb8ab, State: Running, Role: LEADER
I20240430 17:02:34.579687 1285 consensus_queue.cc:237] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.580634 1326 consensus_queue.cc:237] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.584198 1233 raft_consensus.cc:2454] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214687450caa4ed99c39a919d01bb8ab in term 1.
I20240430 17:02:34.590621 982 catalog_manager.cc:5484] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 214687450caa4ed99c39a919d01bb8ab (127.0.103.65). New cstate: current_term: 1 leader_uuid: "214687450caa4ed99c39a919d01bb8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } health_report { overall_health: UNKNOWN } } }
I20240430 17:02:34.602008 982 catalog_manager.cc:5484] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 214687450caa4ed99c39a919d01bb8ab (127.0.103.65). New cstate: current_term: 1 leader_uuid: "214687450caa4ed99c39a919d01bb8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } health_report { overall_health: UNKNOWN } } }
I20240430 17:02:34.606597 1326 raft_consensus.cc:482] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.607141 1326 raft_consensus.cc:504] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.609323 1326 leader_election.cc:288] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317), 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:34.609926 1160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f89981d63b9449a9722b41db39364bd" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de" is_pre_election: true
I20240430 17:02:34.610206 1233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f89981d63b9449a9722b41db39364bd" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c742cba1854d489581f7f53ebc35a2" is_pre_election: true
I20240430 17:02:34.610522 1160 raft_consensus.cc:2454] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214687450caa4ed99c39a919d01bb8ab in term 0.
I20240430 17:02:34.610874 1233 raft_consensus.cc:2454] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214687450caa4ed99c39a919d01bb8ab in term 0.
I20240430 17:02:34.611433 1050 leader_election.cc:302] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 9a037f7ba58a41be985aa8ea2b9198de; no voters:
I20240430 17:02:34.612010 1326 raft_consensus.cc:2781] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:02:34.612286 1326 raft_consensus.cc:482] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:34.612586 1326 raft_consensus.cc:3037] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.616803 1326 raft_consensus.cc:504] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.618615 1326 leader_election.cc:288] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 election: Requested vote from peers 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317), 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:34.618844 1160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f89981d63b9449a9722b41db39364bd" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de"
I20240430 17:02:34.619455 1160 raft_consensus.cc:3037] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.619578 1233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f89981d63b9449a9722b41db39364bd" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c742cba1854d489581f7f53ebc35a2"
I20240430 17:02:34.620077 1233 raft_consensus.cc:3037] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.623795 1160 raft_consensus.cc:2454] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214687450caa4ed99c39a919d01bb8ab in term 1.
I20240430 17:02:34.624050 1233 raft_consensus.cc:2454] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214687450caa4ed99c39a919d01bb8ab in term 1.
I20240430 17:02:34.624717 1050 leader_election.cc:302] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 9a037f7ba58a41be985aa8ea2b9198de; no voters:
I20240430 17:02:34.625339 1326 raft_consensus.cc:2781] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:34.625782 1326 raft_consensus.cc:686] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 1 LEADER]: Becoming Leader. State: Replica: 214687450caa4ed99c39a919d01bb8ab, State: Running, Role: LEADER
I20240430 17:02:34.626497 1326 consensus_queue.cc:237] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.632835 982 catalog_manager.cc:5484] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 214687450caa4ed99c39a919d01bb8ab (127.0.103.65). New cstate: current_term: 1 leader_uuid: "214687450caa4ed99c39a919d01bb8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } health_report { overall_health: UNKNOWN } } }
I20240430 17:02:34.685130 1304 raft_consensus.cc:482] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.685667 1304 raft_consensus.cc:504] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.687348 1304 leader_election.cc:288] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317)
I20240430 17:02:34.688191 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ac9c523f3744d7928c0d14a0625eef" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab" is_pre_election: true
I20240430 17:02:34.688573 1160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ac9c523f3744d7928c0d14a0625eef" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de" is_pre_election: true
I20240430 17:02:34.688894 1084 raft_consensus.cc:2454] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98c742cba1854d489581f7f53ebc35a2 in term 0.
I20240430 17:02:34.689217 1160 raft_consensus.cc:2454] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98c742cba1854d489581f7f53ebc35a2 in term 0.
I20240430 17:02:34.689922 1200 leader_election.cc:302] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 98c742cba1854d489581f7f53ebc35a2; no voters:
I20240430 17:02:34.690531 1304 raft_consensus.cc:2781] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:02:34.690840 1304 raft_consensus.cc:482] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:34.691074 1304 raft_consensus.cc:3037] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.695398 1304 raft_consensus.cc:504] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.696910 1304 leader_election.cc:288] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 election: Requested vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317)
I20240430 17:02:34.697829 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ac9c523f3744d7928c0d14a0625eef" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214687450caa4ed99c39a919d01bb8ab"
I20240430 17:02:34.697996 1160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ac9c523f3744d7928c0d14a0625eef" candidate_uuid: "98c742cba1854d489581f7f53ebc35a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de"
I20240430 17:02:34.698285 1084 raft_consensus.cc:3037] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.698495 1160 raft_consensus.cc:3037] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.702558 1084 raft_consensus.cc:2454] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98c742cba1854d489581f7f53ebc35a2 in term 1.
I20240430 17:02:34.702554 1160 raft_consensus.cc:2454] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98c742cba1854d489581f7f53ebc35a2 in term 1.
I20240430 17:02:34.703665 1200 leader_election.cc:302] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 98c742cba1854d489581f7f53ebc35a2; no voters:
I20240430 17:02:34.704329 1304 raft_consensus.cc:2781] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:34.704741 1304 raft_consensus.cc:686] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Becoming Leader. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Running, Role: LEADER
I20240430 17:02:34.705400 1304 consensus_queue.cc:237] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.712791 982 catalog_manager.cc:5484] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "98c742cba1854d489581f7f53ebc35a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } health_report { overall_health: HEALTHY } } }
I20240430 17:02:34.861330 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:34.867571 1332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:34.868324 1333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:34.869074 1335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:34.869863 413 server_base.cc:1034] running on GCE node
I20240430 17:02:34.870618 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:34.870810 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:34.870944 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496554870932 us; error 0 us; skew 500 ppm
I20240430 17:02:34.871433 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:34.873746 413 webserver.cc:460] Webserver started at http://127.0.103.68:43625/ using document root <none> and password file <none>
I20240430 17:02:34.874154 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:34.874303 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:34.874617 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:34.875865 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-3-root/instance:
uuid: "dad4ee3e8f0049a1bcf3f84220a159cc"
format_stamp: "Formatted at 2024-04-30 17:02:34 on dist-test-slave-x21r"
I20240430 17:02:34.879962 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240430 17:02:34.882943 1340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:34.883708 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240430 17:02:34.883985 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-3-root
uuid: "dad4ee3e8f0049a1bcf3f84220a159cc"
format_stamp: "Formatted at 2024-04-30 17:02:34 on dist-test-slave-x21r"
I20240430 17:02:34.884276 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714496547126677-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:34.900377 1304 consensus_queue.cc:1035] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:34.907363 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:34.908185 1302 consensus_queue.cc:1035] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:34.909356 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:34.913575 1326 raft_consensus.cc:482] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:34.914224 1326 raft_consensus.cc:504] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:34.916759 1326 leader_election.cc:288] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317), 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:34.919055 1159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520fbc131a4b43b7bb23233e1059c51a" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de" is_pre_election: true
I20240430 17:02:34.920177 1159 raft_consensus.cc:2379] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 214687450caa4ed99c39a919d01bb8ab in current term 1: Already voted for candidate 98c742cba1854d489581f7f53ebc35a2 in this term.
I20240430 17:02:34.922550 1233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520fbc131a4b43b7bb23233e1059c51a" candidate_uuid: "214687450caa4ed99c39a919d01bb8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c742cba1854d489581f7f53ebc35a2" is_pre_election: true
I20240430 17:02:34.924963 1050 leader_election.cc:302] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab; no voters: 98c742cba1854d489581f7f53ebc35a2, 9a037f7ba58a41be985aa8ea2b9198de
I20240430 17:02:34.926239 1326 raft_consensus.cc:3037] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:34.930474 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:34.939513 1326 raft_consensus.cc:2726] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 17:02:34.945442 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:34.945781 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:34.946076 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:34.946301 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:34.954380 1296 consensus_queue.cc:1035] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:34.968888 1296 consensus_queue.cc:1035] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [LEADER]: Connected to new peer: Peer: permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:34.975529 1302 consensus_queue.cc:1035] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 17:02:34.980876 1106 debug-util.cc:398] Leaking SignalData structure 0x7b08000aa380 after lost signal to thread 416
W20240430 17:02:34.981654 1106 debug-util.cc:398] Leaking SignalData structure 0x7b08000aaa60 after lost signal to thread 1017
I20240430 17:02:34.983553 1302 consensus_queue.cc:1035] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:34.995175 1326 consensus_queue.cc:1035] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:35.009280 1289 consensus_queue.cc:1035] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:35.036348 1304 consensus_queue.cc:1035] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:35.042975 1328 consensus_queue.cc:1035] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 17:02:35.109246 1285 consensus_queue.cc:1035] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:35.122735 1326 consensus_queue.cc:1035] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 17:02:35.166209 1304 consensus_queue.cc:1035] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:35.176182 1328 consensus_queue.cc:1035] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:35.181746 1304 consensus_queue.cc:1035] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:35.207930 1328 consensus_queue.cc:1035] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:35.269148 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:35643
I20240430 17:02:35.269196 1423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.68:35643 every 8 connection(s)
I20240430 17:02:35.285190 1424 heartbeater.cc:347] Connected to a master server at 127.0.103.126:40275
I20240430 17:02:35.285619 1424 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:35.286391 1424 heartbeater.cc:511] Master 127.0.103.126:40275 requested a full tablet report, sending...
I20240430 17:02:35.288259 980 ts_manager.cc:194] Registered new tserver with Master: dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68:35643)
I20240430 17:02:35.290038 980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54758
I20240430 17:02:36.292109 1424 heartbeater.cc:503] Master 127.0.103.126:40275 was elected leader, sending a full tablet report...
I20240430 17:02:36.333292 1036 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:02:36.351253 1233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "19ae7aeb77f14de988eb80a85aa7ea45"
dest_uuid: "98c742cba1854d489581f7f53ebc35a2"
mode: GRACEFUL
new_leader_uuid: "9a037f7ba58a41be985aa8ea2b9198de"
from {username='slave'} at 127.0.0.1:57962
I20240430 17:02:36.351852 1233 raft_consensus.cc:595] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Received request to transfer leadership to TS 9a037f7ba58a41be985aa8ea2b9198de
I20240430 17:02:36.354133 1233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "520fbc131a4b43b7bb23233e1059c51a"
dest_uuid: "98c742cba1854d489581f7f53ebc35a2"
mode: GRACEFUL
new_leader_uuid: "9a037f7ba58a41be985aa8ea2b9198de"
from {username='slave'} at 127.0.0.1:57962
I20240430 17:02:36.354640 1233 raft_consensus.cc:595] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Received request to transfer leadership to TS 9a037f7ba58a41be985aa8ea2b9198de
I20240430 17:02:36.355530 1036 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240430 17:02:36.356006 1036 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:02:36.474424 1302 raft_consensus.cc:979] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2: : Instructing follower 9a037f7ba58a41be985aa8ea2b9198de to start an election
I20240430 17:02:36.474970 1302 raft_consensus.cc:1067] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Signalling peer 9a037f7ba58a41be985aa8ea2b9198de to start an election
I20240430 17:02:36.476644 1158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "520fbc131a4b43b7bb23233e1059c51a"
dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de"
from {username='slave'} at 127.0.0.1:41734
I20240430 17:02:36.477208 1158 raft_consensus.cc:482] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240430 17:02:36.477497 1158 raft_consensus.cc:3037] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Advancing to term 2
I20240430 17:02:36.482124 1158 raft_consensus.cc:504] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:36.484869 1085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520fbc131a4b43b7bb23233e1059c51a" candidate_uuid: "9a037f7ba58a41be985aa8ea2b9198de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "214687450caa4ed99c39a919d01bb8ab"
I20240430 17:02:36.485093 1158 leader_election.cc:288] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [CANDIDATE]: Term 2 election: Requested vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:36.485105 1233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520fbc131a4b43b7bb23233e1059c51a" candidate_uuid: "9a037f7ba58a41be985aa8ea2b9198de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "98c742cba1854d489581f7f53ebc35a2"
I20240430 17:02:36.485574 1085 raft_consensus.cc:3037] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Advancing to term 2
I20240430 17:02:36.485905 1233 raft_consensus.cc:3032] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 17:02:36.486343 1233 raft_consensus.cc:729] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Running, Role: LEADER
I20240430 17:02:36.487097 1233 consensus_queue.cc:260] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:36.488023 1233 raft_consensus.cc:3037] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Advancing to term 2
I20240430 17:02:36.492488 1233 raft_consensus.cc:2454] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a037f7ba58a41be985aa8ea2b9198de in term 2.
I20240430 17:02:36.492767 1085 raft_consensus.cc:2454] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a037f7ba58a41be985aa8ea2b9198de in term 2.
I20240430 17:02:36.493433 1125 leader_election.cc:302] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98c742cba1854d489581f7f53ebc35a2, 9a037f7ba58a41be985aa8ea2b9198de; no voters:
I20240430 17:02:36.494187 1429 raft_consensus.cc:2781] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 2 FOLLOWER]: Leader election won for term 2
I20240430 17:02:36.495340 1429 raft_consensus.cc:686] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 2 LEADER]: Becoming Leader. State: Replica: 9a037f7ba58a41be985aa8ea2b9198de, State: Running, Role: LEADER
I20240430 17:02:36.496093 1429 consensus_queue.cc:237] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:36.504503 980 catalog_manager.cc:5484] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de reported cstate change: term changed from 1 to 2, leader changed from 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67) to 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66). New cstate: current_term: 2 leader_uuid: "9a037f7ba58a41be985aa8ea2b9198de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } health_report { overall_health: UNKNOWN } } }
I20240430 17:02:36.544013 1289 raft_consensus.cc:979] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2: : Instructing follower 9a037f7ba58a41be985aa8ea2b9198de to start an election
I20240430 17:02:36.544461 1302 raft_consensus.cc:1067] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Signalling peer 9a037f7ba58a41be985aa8ea2b9198de to start an election
I20240430 17:02:36.546032 1158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19ae7aeb77f14de988eb80a85aa7ea45"
dest_uuid: "9a037f7ba58a41be985aa8ea2b9198de"
from {username='slave'} at 127.0.0.1:41734
I20240430 17:02:36.546526 1158 raft_consensus.cc:482] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240430 17:02:36.546813 1158 raft_consensus.cc:3037] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Advancing to term 2
I20240430 17:02:36.551931 1158 raft_consensus.cc:504] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:36.553507 1158 leader_election.cc:288] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [CANDIDATE]: Term 2 election: Requested vote from peers 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233), 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:36.554353 1085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ae7aeb77f14de988eb80a85aa7ea45" candidate_uuid: "9a037f7ba58a41be985aa8ea2b9198de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "214687450caa4ed99c39a919d01bb8ab"
I20240430 17:02:36.554626 1233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ae7aeb77f14de988eb80a85aa7ea45" candidate_uuid: "9a037f7ba58a41be985aa8ea2b9198de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "98c742cba1854d489581f7f53ebc35a2"
I20240430 17:02:36.554917 1085 raft_consensus.cc:3037] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Advancing to term 2
I20240430 17:02:36.555353 1233 raft_consensus.cc:3032] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 17:02:36.555716 1233 raft_consensus.cc:729] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 98c742cba1854d489581f7f53ebc35a2, State: Running, Role: LEADER
I20240430 17:02:36.556500 1233 consensus_queue.cc:260] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:36.557827 1233 raft_consensus.cc:3037] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Advancing to term 2
I20240430 17:02:36.559841 1085 raft_consensus.cc:2454] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a037f7ba58a41be985aa8ea2b9198de in term 2.
I20240430 17:02:36.561308 1126 leader_election.cc:302] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214687450caa4ed99c39a919d01bb8ab, 9a037f7ba58a41be985aa8ea2b9198de; no voters:
I20240430 17:02:36.562160 1429 raft_consensus.cc:2781] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 2 FOLLOWER]: Leader election won for term 2
I20240430 17:02:36.562623 1429 raft_consensus.cc:686] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 2 LEADER]: Becoming Leader. State: Replica: 9a037f7ba58a41be985aa8ea2b9198de, State: Running, Role: LEADER
I20240430 17:02:36.563258 1429 consensus_queue.cc:237] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } }
I20240430 17:02:36.568248 1233 raft_consensus.cc:2454] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a037f7ba58a41be985aa8ea2b9198de in term 2.
I20240430 17:02:36.570576 980 catalog_manager.cc:5484] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de reported cstate change: term changed from 1 to 2, leader changed from 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67) to 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66). New cstate: current_term: 2 leader_uuid: "9a037f7ba58a41be985aa8ea2b9198de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } health_report { overall_health: UNKNOWN } } }
I20240430 17:02:36.960129 1085 raft_consensus.cc:1261] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [term 2 FOLLOWER]: Refusing update from remote peer 9a037f7ba58a41be985aa8ea2b9198de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 17:02:36.961548 1429 consensus_queue.cc:1035] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [LEADER]: Connected to new peer: Peer: permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 17:02:36.973325 1233 raft_consensus.cc:1261] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 2 FOLLOWER]: Refusing update from remote peer 9a037f7ba58a41be985aa8ea2b9198de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 17:02:36.975720 1432 consensus_queue.cc:1035] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240430 17:02:37.053606 1085 raft_consensus.cc:1261] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [term 2 FOLLOWER]: Refusing update from remote peer 9a037f7ba58a41be985aa8ea2b9198de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 17:02:37.054749 1441 consensus_queue.cc:1035] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [LEADER]: Connected to new peer: Peer: permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 17:02:37.062850 1233 raft_consensus.cc:1261] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 2 FOLLOWER]: Refusing update from remote peer 9a037f7ba58a41be985aa8ea2b9198de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 17:02:37.066561 1429 consensus_queue.cc:1035] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 17:02:38.357306 1036 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:02:38.366159 1036 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:02:38.366726 1036 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:02:40.367667 1036 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:02:40.373941 1036 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:02:40.374342 1036 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:02:40.375213 1159 consensus_queue.cc:237] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } attrs { replace: true } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } }
I20240430 17:02:40.382705 1234 raft_consensus.cc:1261] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Refusing update from remote peer 9a037f7ba58a41be985aa8ea2b9198de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:02:40.384831 1084 raft_consensus.cc:1261] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Refusing update from remote peer 9a037f7ba58a41be985aa8ea2b9198de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:02:40.384656 1454 consensus_queue.cc:1035] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240430 17:02:40.386384 1448 consensus_queue.cc:1035] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [LEADER]: Connected to new peer: Peer: permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240430 17:02:40.400501 1123 consensus_peers.cc:485] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de -> Peer dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68:35643): Couldn't send request to peer dad4ee3e8f0049a1bcf3f84220a159cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1fd9f27d6394358bee83bfd16b7db68. This is attempt 1: this message will repeat every 5th retry.
I20240430 17:02:40.404070 1454 raft_consensus.cc:2932] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } attrs { replace: true } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.406142 1084 raft_consensus.cc:2932] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } attrs { replace: true } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.407297 1234 raft_consensus.cc:2932] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } attrs { replace: true } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.419692 982 catalog_manager.cc:5484] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 reported cstate change: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "9a037f7ba58a41be985aa8ea2b9198de" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } attrs { replace: true } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.444224 1084 consensus_queue.cc:237] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } attrs { replace: true } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } }
I20240430 17:02:40.452466 1159 raft_consensus.cc:1261] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Refusing update from remote peer 214687450caa4ed99c39a919d01bb8ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:02:40.452626 1234 raft_consensus.cc:1261] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Refusing update from remote peer 214687450caa4ed99c39a919d01bb8ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:02:40.454002 1469 consensus_queue.cc:1035] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240430 17:02:40.455009 1457 consensus_queue.cc:1035] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240430 17:02:40.463721 1048 consensus_peers.cc:485] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab -> Peer dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68:35643): Couldn't send request to peer dad4ee3e8f0049a1bcf3f84220a159cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f89981d63b9449a9722b41db39364bd. This is attempt 1: this message will repeat every 5th retry.
I20240430 17:02:40.465623 1457 raft_consensus.cc:2932] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } attrs { replace: true } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.467187 1159 raft_consensus.cc:2932] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } attrs { replace: true } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.468174 1234 raft_consensus.cc:2932] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } attrs { replace: true } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.477102 1084 consensus_queue.cc:237] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } }
I20240430 17:02:40.477460 981 catalog_manager.cc:5484] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab reported cstate change: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "214687450caa4ed99c39a919d01bb8ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240430 17:02:40.487332 1234 raft_consensus.cc:1261] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Refusing update from remote peer 214687450caa4ed99c39a919d01bb8ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240430 17:02:40.487695 1048 consensus_peers.cc:485] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab -> Peer dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68:35643): Couldn't send request to peer dad4ee3e8f0049a1bcf3f84220a159cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 640402773e2947058566ccd9f794166c. This is attempt 1: this message will repeat every 5th retry.
I20240430 17:02:40.487771 1159 raft_consensus.cc:1261] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Refusing update from remote peer 214687450caa4ed99c39a919d01bb8ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:02:40.488776 1457 consensus_queue.cc:1035] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 17:02:40.489660 1469 consensus_queue.cc:1035] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 17:02:40.498979 1469 raft_consensus.cc:2932] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.500873 1159 raft_consensus.cc:2932] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.502033 1234 raft_consensus.cc:2932] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.513828 980 catalog_manager.cc:5484] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de reported cstate change: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "214687450caa4ed99c39a919d01bb8ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.526487 1234 consensus_queue.cc:237] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } }
I20240430 17:02:40.533268 1084 raft_consensus.cc:1261] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Refusing update from remote peer 98c742cba1854d489581f7f53ebc35a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:02:40.535951 1449 consensus_queue.cc:1035] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240430 17:02:40.538784 1159 raft_consensus.cc:1261] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Refusing update from remote peer 98c742cba1854d489581f7f53ebc35a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:02:40.540052 1479 consensus_queue.cc:1035] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240430 17:02:40.545321 1449 raft_consensus.cc:2932] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
W20240430 17:02:40.547531 1197 consensus_peers.cc:485] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 -> Peer dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68:35643): Couldn't send request to peer dad4ee3e8f0049a1bcf3f84220a159cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d331ac9b637247048e3915e06aaf2a25. This is attempt 1: this message will repeat every 5th retry.
I20240430 17:02:40.547405 1084 raft_consensus.cc:2932] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.551146 1159 raft_consensus.cc:2932] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.560333 981 catalog_manager.cc:5484] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab reported cstate change: config changed from index -1 to 2, NON_VOTER dad4ee3e8f0049a1bcf3f84220a159cc (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "98c742cba1854d489581f7f53ebc35a2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } } }
I20240430 17:02:40.763509 1484 ts_tablet_manager.cc:902] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc: Initiating tablet copy from peer 9a037f7ba58a41be985aa8ea2b9198de (127.0.103.66:43317)
I20240430 17:02:40.767980 1484 tablet_copy_client.cc:308] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43317
I20240430 17:02:40.778884 1170 tablet_copy_service.cc:140] P 9a037f7ba58a41be985aa8ea2b9198de: Received BeginTabletCopySession request for tablet d1fd9f27d6394358bee83bfd16b7db68 from peer dad4ee3e8f0049a1bcf3f84220a159cc ({username='slave'} at 127.0.0.1:48672)
I20240430 17:02:40.779449 1170 tablet_copy_service.cc:161] P 9a037f7ba58a41be985aa8ea2b9198de: Beginning new tablet copy session on tablet d1fd9f27d6394358bee83bfd16b7db68 from peer dad4ee3e8f0049a1bcf3f84220a159cc at {username='slave'} at 127.0.0.1:48672: session id = dad4ee3e8f0049a1bcf3f84220a159cc-d1fd9f27d6394358bee83bfd16b7db68
I20240430 17:02:40.786407 1170 tablet_copy_source_session.cc:205] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de: Tablet Copy: opened 0 blocks and 1 log segments
I20240430 17:02:40.789683 1484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1fd9f27d6394358bee83bfd16b7db68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:40.791250 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
W20240430 17:02:40.794921 970 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 7 similar messages]
W20240430 17:02:40.799777 1031 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240430 17:02:40.800100 1031 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240430 17:02:40.808362 1031 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240430 17:02:40.808779 1031 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240430 17:02:40.809577 1484 tablet_copy_client.cc:783] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc: tablet copy: Starting download of 0 data blocks...
I20240430 17:02:40.810309 1484 tablet_copy_client.cc:647] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc: tablet copy: Starting download of 1 WAL segments...
I20240430 17:02:40.818675 1484 tablet_copy_client.cc:523] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240430 17:02:40.827806 1484 tablet_bootstrap.cc:492] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc: Bootstrap starting.
I20240430 17:02:40.843881 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240430 17:02:40.845108 1200 consensus_peers.cc:485] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 -> Peer 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233): Couldn't send request to peer 214687450caa4ed99c39a919d01bb8ab. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240430 17:02:40.845240 413 tablet_replica.cc:330] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab: stopping tablet replica
I20240430 17:02:40.846312 413 raft_consensus.cc:2229] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:40.846889 413 raft_consensus.cc:2258] T 69ac9c523f3744d7928c0d14a0625eef P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:40.849380 413 tablet_replica.cc:330] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab: stopping tablet replica
I20240430 17:02:40.849812 413 raft_consensus.cc:2229] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:40.850785 413 raft_consensus.cc:2258] T 743ddfaf92da4b2896d8146ffe5560ae P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:40.852810 1484 tablet_bootstrap.cc:492] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240430 17:02:40.852895 413 tablet_replica.cc:330] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab: stopping tablet replica
I20240430 17:02:40.853747 413 raft_consensus.cc:2229] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:40.853827 1484 tablet_bootstrap.cc:492] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc: Bootstrap complete.
I20240430 17:02:40.855429 1484 ts_tablet_manager.cc:1367] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.002s
I20240430 17:02:40.855610 413 raft_consensus.cc:2258] T 9f89981d63b9449a9722b41db39364bd P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:40.858419 413 tablet_replica.cc:330] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab: stopping tablet replica
I20240430 17:02:40.858510 1484 raft_consensus.cc:348] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } attrs { replace: true } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } }
I20240430 17:02:40.859380 413 raft_consensus.cc:2229] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:40.859477 1484 raft_consensus.cc:729] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dad4ee3e8f0049a1bcf3f84220a159cc, State: Initialized, Role: LEARNER
I20240430 17:02:40.859995 413 raft_consensus.cc:2258] T d1fd9f27d6394358bee83bfd16b7db68 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:40.860232 1484 consensus_queue.cc:260] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } attrs { replace: true } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } } peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } }
I20240430 17:02:40.863364 413 tablet_replica.cc:330] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab: stopping tablet replica
I20240430 17:02:40.864135 413 raft_consensus.cc:2229] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:40.864706 413 raft_consensus.cc:2258] T d331ac9b637247048e3915e06aaf2a25 P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:40.867595 413 tablet_replica.cc:330] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab: stopping tablet replica
I20240430 17:02:40.868675 413 raft_consensus.cc:2229] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:40.869220 413 raft_consensus.cc:2258] T 520fbc131a4b43b7bb23233e1059c51a P 214687450caa4ed99c39a919d01bb8ab [term 2 FOLLOWER]: Raft consensus is shut down!
W20240430 17:02:40.870384 1126 consensus_peers.cc:485] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de -> Peer 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233): Couldn't send request to peer 214687450caa4ed99c39a919d01bb8ab. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240430 17:02:40.871642 413 tablet_replica.cc:330] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab: stopping tablet replica
I20240430 17:02:40.872347 413 raft_consensus.cc:2229] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:40.872429 1484 ts_tablet_manager.cc:1398] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240430 17:02:40.873562 413 raft_consensus.cc:2258] T 640402773e2947058566ccd9f794166c P 214687450caa4ed99c39a919d01bb8ab [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 17:02:40.874181 1126 consensus_peers.cc:485] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de -> Peer 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233): Couldn't send request to peer 214687450caa4ed99c39a919d01bb8ab. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240430 17:02:40.875075 1170 tablet_copy_service.cc:342] P 9a037f7ba58a41be985aa8ea2b9198de: Request end of tablet copy session dad4ee3e8f0049a1bcf3f84220a159cc-d1fd9f27d6394358bee83bfd16b7db68 received from {username='slave'} at 127.0.0.1:48672
I20240430 17:02:40.875546 1170 tablet_copy_service.cc:433] P 9a037f7ba58a41be985aa8ea2b9198de: ending tablet copy session dad4ee3e8f0049a1bcf3f84220a159cc-d1fd9f27d6394358bee83bfd16b7db68 on tablet d1fd9f27d6394358bee83bfd16b7db68 with peer dad4ee3e8f0049a1bcf3f84220a159cc
I20240430 17:02:40.876127 413 tablet_replica.cc:330] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab: stopping tablet replica
I20240430 17:02:40.877292 413 raft_consensus.cc:2229] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:40.878010 413 raft_consensus.cc:2258] T 19ae7aeb77f14de988eb80a85aa7ea45 P 214687450caa4ed99c39a919d01bb8ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:40.943754 413 tablet_server.cc:194] TabletServer@127.0.103.65:0 shutdown complete.
W20240430 17:02:40.950310 1126 consensus_peers.cc:485] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de -> Peer 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233): Couldn't send request to peer 214687450caa4ed99c39a919d01bb8ab. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 17:02:40.966953 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
W20240430 17:02:40.977569 1031 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240430 17:02:40.977968 1031 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240430 17:02:40.996950 1200 consensus_peers.cc:485] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 -> Peer 214687450caa4ed99c39a919d01bb8ab (127.0.103.65:38233): Couldn't send request to peer 214687450caa4ed99c39a919d01bb8ab. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 17:02:41.002905 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:02:41.003793 413 tablet_replica.cc:330] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de: stopping tablet replica
I20240430 17:02:41.004774 413 raft_consensus.cc:2229] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:41.005312 413 raft_consensus.cc:2258] T 9f89981d63b9449a9722b41db39364bd P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.008080 413 tablet_replica.cc:330] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de: stopping tablet replica
I20240430 17:02:41.008589 413 raft_consensus.cc:2229] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:41.008960 413 raft_consensus.cc:2258] T 69ac9c523f3744d7928c0d14a0625eef P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.010586 413 tablet_replica.cc:330] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de: stopping tablet replica
I20240430 17:02:41.011046 413 raft_consensus.cc:2229] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:41.011472 413 raft_consensus.cc:2258] T d331ac9b637247048e3915e06aaf2a25 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.013748 413 tablet_replica.cc:330] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de: stopping tablet replica
I20240430 17:02:41.014207 413 raft_consensus.cc:2229] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:41.014577 413 raft_consensus.cc:2258] T 743ddfaf92da4b2896d8146ffe5560ae P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.016310 413 tablet_replica.cc:330] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de: stopping tablet replica
I20240430 17:02:41.016754 413 raft_consensus.cc:2229] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 2 LEADER]: Raft consensus shutting down.
I20240430 17:02:41.017436 413 raft_consensus.cc:2258] T 520fbc131a4b43b7bb23233e1059c51a P 9a037f7ba58a41be985aa8ea2b9198de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.019049 413 tablet_replica.cc:330] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de: stopping tablet replica
I20240430 17:02:41.019600 413 raft_consensus.cc:2229] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 2 LEADER]: Raft consensus shutting down.
I20240430 17:02:41.020201 413 raft_consensus.cc:2258] T 19ae7aeb77f14de988eb80a85aa7ea45 P 9a037f7ba58a41be985aa8ea2b9198de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.021948 413 tablet_replica.cc:330] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de: stopping tablet replica
I20240430 17:02:41.022375 413 raft_consensus.cc:2229] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:41.022835 413 raft_consensus.cc:2258] T 640402773e2947058566ccd9f794166c P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.024998 1484 ts_tablet_manager.cc:902] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc: Initiating tablet copy from peer 98c742cba1854d489581f7f53ebc35a2 (127.0.103.67:34779)
I20240430 17:02:41.025805 413 tablet_replica.cc:330] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de: stopping tablet replica
I20240430 17:02:41.026273 413 raft_consensus.cc:2229] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:41.027364 413 raft_consensus.cc:2258] T d1fd9f27d6394358bee83bfd16b7db68 P 9a037f7ba58a41be985aa8ea2b9198de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.029443 1484 tablet_copy_client.cc:308] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:34779
I20240430 17:02:41.050213 1244 tablet_copy_service.cc:140] P 98c742cba1854d489581f7f53ebc35a2: Received BeginTabletCopySession request for tablet d331ac9b637247048e3915e06aaf2a25 from peer dad4ee3e8f0049a1bcf3f84220a159cc ({username='slave'} at 127.0.0.1:32940)
I20240430 17:02:41.050657 1244 tablet_copy_service.cc:161] P 98c742cba1854d489581f7f53ebc35a2: Beginning new tablet copy session on tablet d331ac9b637247048e3915e06aaf2a25 from peer dad4ee3e8f0049a1bcf3f84220a159cc at {username='slave'} at 127.0.0.1:32940: session id = dad4ee3e8f0049a1bcf3f84220a159cc-d331ac9b637247048e3915e06aaf2a25
I20240430 17:02:41.056444 1244 tablet_copy_source_session.cc:205] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2: Tablet Copy: opened 0 blocks and 1 log segments
I20240430 17:02:41.059826 413 tablet_server.cc:194] TabletServer@127.0.103.66:0 shutdown complete.
I20240430 17:02:41.059876 1484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d331ac9b637247048e3915e06aaf2a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:41.073175 1484 tablet_copy_client.cc:783] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc: tablet copy: Starting download of 0 data blocks...
I20240430 17:02:41.073824 1484 tablet_copy_client.cc:647] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc: tablet copy: Starting download of 1 WAL segments...
I20240430 17:02:41.077787 1484 tablet_copy_client.cc:523] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240430 17:02:41.083774 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20240430 17:02:41.084537 1484 tablet_bootstrap.cc:492] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc: Bootstrap starting.
W20240430 17:02:41.089726 1031 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240430 17:02:41.090086 1031 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240430 17:02:41.097412 413 tablet_copy_service.cc:362] P 98c742cba1854d489581f7f53ebc35a2: Destroying tablet copy session dad4ee3e8f0049a1bcf3f84220a159cc-d331ac9b637247048e3915e06aaf2a25 due to service shutdown
I20240430 17:02:41.097755 413 tablet_copy_service.cc:433] P 98c742cba1854d489581f7f53ebc35a2: ending tablet copy session dad4ee3e8f0049a1bcf3f84220a159cc-d331ac9b637247048e3915e06aaf2a25 on tablet d331ac9b637247048e3915e06aaf2a25 with peer dad4ee3e8f0049a1bcf3f84220a159cc
I20240430 17:02:41.113011 1484 tablet_bootstrap.cc:492] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240430 17:02:41.113960 1484 tablet_bootstrap.cc:492] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc: Bootstrap complete.
I20240430 17:02:41.114373 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:02:41.114754 1484 ts_tablet_manager.cc:1367] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.004s
I20240430 17:02:41.115290 413 tablet_replica.cc:330] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2: stopping tablet replica
I20240430 17:02:41.116119 413 raft_consensus.cc:2229] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:41.117118 413 raft_consensus.cc:2258] T 69ac9c523f3744d7928c0d14a0625eef P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.117362 1484 raft_consensus.cc:348] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } }
I20240430 17:02:41.118257 1484 raft_consensus.cc:729] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dad4ee3e8f0049a1bcf3f84220a159cc, State: Initialized, Role: LEARNER
I20240430 17:02:41.118885 1484 consensus_queue.cc:260] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "98c742cba1854d489581f7f53ebc35a2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34779 } } peers { permanent_uuid: "9a037f7ba58a41be985aa8ea2b9198de" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43317 } attrs { replace: true } } peers { permanent_uuid: "214687450caa4ed99c39a919d01bb8ab" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38233 } } peers { permanent_uuid: "dad4ee3e8f0049a1bcf3f84220a159cc" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35643 } attrs { promote: true } }
I20240430 17:02:41.119588 413 tablet_replica.cc:330] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2: stopping tablet replica
I20240430 17:02:41.120345 413 raft_consensus.cc:2229] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:41.120889 413 raft_consensus.cc:2258] T 9f89981d63b9449a9722b41db39364bd P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.121153 1484 ts_tablet_manager.cc:1398] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240430 17:02:41.123339 413 tablet_replica.cc:330] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2: stopping tablet replica
I20240430 17:02:41.124037 413 raft_consensus.cc:2229] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:41.124534 413 raft_consensus.cc:2258] T 743ddfaf92da4b2896d8146ffe5560ae P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.126668 413 tablet_replica.cc:330] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2: stopping tablet replica
I20240430 17:02:41.127250 413 raft_consensus.cc:2229] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:41.127653 413 raft_consensus.cc:2258] T 520fbc131a4b43b7bb23233e1059c51a P 98c742cba1854d489581f7f53ebc35a2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.129719 413 tablet_replica.cc:330] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2: stopping tablet replica
I20240430 17:02:41.130376 413 raft_consensus.cc:2229] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:41.130870 413 raft_consensus.cc:2258] T 640402773e2947058566ccd9f794166c P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.133622 413 tablet_replica.cc:330] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2: stopping tablet replica
I20240430 17:02:41.134119 413 raft_consensus.cc:2229] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:41.134478 413 raft_consensus.cc:2258] T 19ae7aeb77f14de988eb80a85aa7ea45 P 98c742cba1854d489581f7f53ebc35a2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.136037 413 tablet_replica.cc:330] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2: stopping tablet replica
I20240430 17:02:41.136456 413 raft_consensus.cc:2229] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:41.137228 413 raft_consensus.cc:2258] T d331ac9b637247048e3915e06aaf2a25 P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.138975 413 tablet_replica.cc:330] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2: stopping tablet replica
I20240430 17:02:41.139436 413 raft_consensus.cc:2229] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:41.139817 413 raft_consensus.cc:2258] T d1fd9f27d6394358bee83bfd16b7db68 P 98c742cba1854d489581f7f53ebc35a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.165573 413 tablet_server.cc:194] TabletServer@127.0.103.67:0 shutdown complete.
W20240430 17:02:41.169760 1484 tablet_copy_client.cc:1108] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.0.103.67:34779: connect: Connection refused (error 111)
I20240430 17:02:41.181442 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20240430 17:02:41.203266 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:02:41.204049 413 tablet_replica.cc:330] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc: stopping tablet replica
I20240430 17:02:41.204543 413 raft_consensus.cc:2229] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc [term 1 LEARNER]: Raft consensus shutting down.
I20240430 17:02:41.204910 413 raft_consensus.cc:2258] T d331ac9b637247048e3915e06aaf2a25 P dad4ee3e8f0049a1bcf3f84220a159cc [term 1 LEARNER]: Raft consensus is shut down!
I20240430 17:02:41.206624 413 tablet_replica.cc:330] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc: stopping tablet replica
I20240430 17:02:41.207060 413 raft_consensus.cc:2229] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc [term 1 LEARNER]: Raft consensus shutting down.
I20240430 17:02:41.207463 413 raft_consensus.cc:2258] T d1fd9f27d6394358bee83bfd16b7db68 P dad4ee3e8f0049a1bcf3f84220a159cc [term 1 LEARNER]: Raft consensus is shut down!
I20240430 17:02:41.225656 413 tablet_server.cc:194] TabletServer@127.0.103.68:0 shutdown complete.
I20240430 17:02:41.235512 413 master.cc:537] Master@127.0.103.126:40275 shutting down...
I20240430 17:02:41.251727 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:41.252213 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:41.252512 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P d90eaf52a5e94b91bc5cfe924bc7d017: stopping tablet replica
I20240430 17:02:41.270424 413 master.cc:552] Master@127.0.103.126:40275 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (9113 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240430 17:02:41.309859 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:36315,127.0.103.125:33953,127.0.103.124:41743
I20240430 17:02:41.311154 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:41.317497 1502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:41.318192 1503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:41.319032 1505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:41.319779 413 server_base.cc:1034] running on GCE node
I20240430 17:02:41.320601 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:41.320803 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:41.320955 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496561320937 us; error 0 us; skew 500 ppm
I20240430 17:02:41.321419 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:41.323956 413 webserver.cc:460] Webserver started at http://127.0.103.126:35329/ using document root <none> and password file <none>
I20240430 17:02:41.324407 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:41.324576 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:41.324820 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:41.325850 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-0-root/instance:
uuid: "ca9980152ed94b4395d601f62e7b8c81"
format_stamp: "Formatted at 2024-04-30 17:02:41 on dist-test-slave-x21r"
I20240430 17:02:41.329988 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240430 17:02:41.333045 1510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:41.333840 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240430 17:02:41.334065 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-0-root
uuid: "ca9980152ed94b4395d601f62e7b8c81"
format_stamp: "Formatted at 2024-04-30 17:02:41 on dist-test-slave-x21r"
I20240430 17:02:41.334298 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:41.356734 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:41.357899 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:41.392907 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:36315
I20240430 17:02:41.392990 1561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.126:36315 every 8 connection(s)
I20240430 17:02:41.396701 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240430 17:02:41.396701 1562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 17:02:41.402987 1564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:41.403791 1565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:41.405267 1562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
W20240430 17:02:41.405978 1567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:41.406838 413 server_base.cc:1034] running on GCE node
I20240430 17:02:41.407925 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:41.408138 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:41.408298 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496561408283 us; error 0 us; skew 500 ppm
I20240430 17:02:41.408990 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:41.411573 413 webserver.cc:460] Webserver started at http://127.0.103.125:37469/ using document root <none> and password file <none>
I20240430 17:02:41.412159 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:41.412384 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:41.412689 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:41.413861 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-1-root/instance:
uuid: "a02319a009aa481584f555e8b3d77b24"
format_stamp: "Formatted at 2024-04-30 17:02:41 on dist-test-slave-x21r"
I20240430 17:02:41.418077 1562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } has no permanent_uuid. Determining permanent_uuid...
I20240430 17:02:41.419212 413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
W20240430 17:02:41.423406 1562 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.103.125:33953: Network error: Client connection negotiation failed: client connection to 127.0.103.125:33953: connect: Connection refused (error 111)
I20240430 17:02:41.423811 1575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:41.424664 413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240430 17:02:41.424902 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-1-root
uuid: "a02319a009aa481584f555e8b3d77b24"
format_stamp: "Formatted at 2024-04-30 17:02:41 on dist-test-slave-x21r"
I20240430 17:02:41.425135 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:41.466542 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:41.467738 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:41.472877 1562 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } attempt: 1
W20240430 17:02:41.476815 1562 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.103.125:33953: Network error: Client connection negotiation failed: client connection to 127.0.103.125:33953: connect: Connection refused (error 111)
I20240430 17:02:41.503439 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.125:33953
I20240430 17:02:41.503522 1626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.125:33953 every 8 connection(s)
I20240430 17:02:41.506973 1627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:41.507189 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:41.513509 1629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:41.514328 1630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:41.514894 1627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
W20240430 17:02:41.515460 1632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:41.516613 413 server_base.cc:1034] running on GCE node
I20240430 17:02:41.517506 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:41.517714 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:41.517868 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496561517853 us; error 0 us; skew 500 ppm
I20240430 17:02:41.518347 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:41.521070 413 webserver.cc:460] Webserver started at http://127.0.103.124:39997/ using document root <none> and password file <none>
I20240430 17:02:41.521648 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:41.521883 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:41.522184 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:41.523370 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-2-root/instance:
uuid: "e1ef14b5433f4dbc885454e5b7eb79fa"
format_stamp: "Formatted at 2024-04-30 17:02:41 on dist-test-slave-x21r"
I20240430 17:02:41.527096 1627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } has no permanent_uuid. Determining permanent_uuid...
I20240430 17:02:41.528661 413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240430 17:02:41.532838 1639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:41.533718 413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240430 17:02:41.534041 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-2-root
uuid: "e1ef14b5433f4dbc885454e5b7eb79fa"
format_stamp: "Formatted at 2024-04-30 17:02:41 on dist-test-slave-x21r"
I20240430 17:02:41.534377 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:41.539508 1627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } has no permanent_uuid. Determining permanent_uuid...
I20240430 17:02:41.540314 1562 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } attempt: 2
W20240430 17:02:41.545342 1627 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.103.124:41743: Network error: Client connection negotiation failed: client connection to 127.0.103.124:41743: connect: Connection refused (error 111)
I20240430 17:02:41.550120 1562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } has no permanent_uuid. Determining permanent_uuid...
W20240430 17:02:41.553876 1562 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.103.124:41743: Network error: Client connection negotiation failed: client connection to 127.0.103.124:41743: connect: Connection refused (error 111)
I20240430 17:02:41.564567 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:41.565675 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:41.568848 1627 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } attempt: 1
W20240430 17:02:41.572866 1627 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.103.124:41743: Network error: Client connection negotiation failed: client connection to 127.0.103.124:41743: connect: Connection refused (error 111)
I20240430 17:02:41.602099 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.124:41743
I20240430 17:02:41.602190 1691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.124:41743 every 8 connection(s)
I20240430 17:02:41.604352 1562 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } attempt: 1
I20240430 17:02:41.605078 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240430 17:02:41.606194 1692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:41.614019 1692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
I20240430 17:02:41.622403 1562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81: Bootstrap starting.
I20240430 17:02:41.626031 1692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } has no permanent_uuid. Determining permanent_uuid...
I20240430 17:02:41.627947 1562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:41.632831 1562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81: No bootstrap required, opened a new log
I20240430 17:02:41.634848 1562 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:41.635618 1562 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:41.635610 1692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } has no permanent_uuid. Determining permanent_uuid...
I20240430 17:02:41.636034 1562 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9980152ed94b4395d601f62e7b8c81, State: Initialized, Role: FOLLOWER
I20240430 17:02:41.636708 1562 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:41.638720 1697 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:41.639551 1697 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:41.640178 1562 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 17:02:41.644361 1627 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } attempt: 2
I20240430 17:02:41.655464 1692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa: Bootstrap starting.
I20240430 17:02:41.656905 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240430 17:02:41.660578 1692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:41.660899 1627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24: Bootstrap starting.
W20240430 17:02:41.661316 1708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240430 17:02:41.661679 1708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240430 17:02:41.664924 1692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa: No bootstrap required, opened a new log
I20240430 17:02:41.665393 1627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:41.666833 1692 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:41.667424 1692 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:41.667635 1692 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ef14b5433f4dbc885454e5b7eb79fa, State: Initialized, Role: FOLLOWER
I20240430 17:02:41.668217 1692 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:41.669829 1627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24: No bootstrap required, opened a new log
I20240430 17:02:41.670060 1711 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:41.670922 1711 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:41.671375 1711 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:41.671432 1692 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [sys.catalog]: configured and running, proceeding with master startup.
I20240430 17:02:41.671880 1711 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:41.672549 1627 raft_consensus.cc:348] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:41.673311 1627 raft_consensus.cc:374] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:41.673616 1627 raft_consensus.cc:729] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02319a009aa481584f555e8b3d77b24, State: Initialized, Role: FOLLOWER
I20240430 17:02:41.674450 1627 consensus_queue.cc:260] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:41.675032 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9980152ed94b4395d601f62e7b8c81" is_pre_election: true
I20240430 17:02:41.675980 1537 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ef14b5433f4dbc885454e5b7eb79fa in term 0.
I20240430 17:02:41.677919 1640 leader_election.cc:302] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca9980152ed94b4395d601f62e7b8c81, e1ef14b5433f4dbc885454e5b7eb79fa; no voters:
I20240430 17:02:41.681046 1711 leader_election.cc:288] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca9980152ed94b4395d601f62e7b8c81 (127.0.103.126:36315), a02319a009aa481584f555e8b3d77b24 (127.0.103.125:33953)
I20240430 17:02:41.682276 1711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:02:41.682771 1711 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:41.685431 1711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:41.691768 1714 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:41.693199 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a02319a009aa481584f555e8b3d77b24" is_pre_election: true
I20240430 17:02:41.695641 1602 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ef14b5433f4dbc885454e5b7eb79fa in term 0.
I20240430 17:02:41.696795 1714 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:41.693912 1627 sys_catalog.cc:564] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 17:02:41.698726 1711 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:41.701093 1711 leader_election.cc:288] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [CANDIDATE]: Term 1 election: Requested vote from peers ca9980152ed94b4395d601f62e7b8c81 (127.0.103.126:36315), a02319a009aa481584f555e8b3d77b24 (127.0.103.125:33953)
I20240430 17:02:41.703229 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9980152ed94b4395d601f62e7b8c81"
I20240430 17:02:41.703254 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a02319a009aa481584f555e8b3d77b24"
I20240430 17:02:41.704349 1537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:41.704571 1602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:41.714735 1537 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ef14b5433f4dbc885454e5b7eb79fa in term 1.
I20240430 17:02:41.716454 1640 leader_election.cc:302] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca9980152ed94b4395d601f62e7b8c81, e1ef14b5433f4dbc885454e5b7eb79fa; no voters:
I20240430 17:02:41.717576 1711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:41.718219 1711 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 1 LEADER]: Becoming Leader. State: Replica: e1ef14b5433f4dbc885454e5b7eb79fa, State: Running, Role: LEADER
I20240430 17:02:41.719317 1711 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:41.714735 1602 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ef14b5433f4dbc885454e5b7eb79fa in term 1.
I20240430 17:02:41.727847 1712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ef14b5433f4dbc885454e5b7eb79fa. Latest consensus state: current_term: 1 leader_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:41.728914 1712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [sys.catalog]: This master's current role is: LEADER
I20240430 17:02:41.731513 1733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 17:02:41.736864 1732 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240430 17:02:41.738845 1733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 17:02:41.746138 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240430 17:02:41.748013 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240430 17:02:41.749363 1537 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 1 FOLLOWER]: Refusing update from remote peer e1ef14b5433f4dbc885454e5b7eb79fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:02:41.750974 1711 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 17:02:41.751653 1738 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240430 17:02:41.752084 1738 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240430 17:02:41.752676 1602 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 1 FOLLOWER]: Refusing update from remote peer e1ef14b5433f4dbc885454e5b7eb79fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:02:41.754637 1711 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 17:02:41.760036 1739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:41.763916 1740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:41.763185 1697 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ef14b5433f4dbc885454e5b7eb79fa. Latest consensus state: current_term: 1 leader_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:41.765067 1697 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:41.775827 1697 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:41.776748 1697 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [sys.catalog]: This master's current role is: FOLLOWER
W20240430 17:02:41.777058 1745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:41.779935 1714 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ef14b5433f4dbc885454e5b7eb79fa. Latest consensus state: current_term: 1 leader_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:41.780733 1714 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:41.780299 1711 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:41.781291 1711 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [sys.catalog]: This master's current role is: LEADER
I20240430 17:02:41.783013 1733 catalog_manager.cc:1347] Generated new cluster ID: c2d7a0ed4a774a6887ae7d288fbf8152
I20240430 17:02:41.783377 1733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 17:02:41.783914 1712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:41.784727 1712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [sys.catalog]: This master's current role is: LEADER
I20240430 17:02:41.783928 413 server_base.cc:1034] running on GCE node
I20240430 17:02:41.785956 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:41.786218 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:41.786409 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496561786393 us; error 0 us; skew 500 ppm
I20240430 17:02:41.787088 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:41.790468 413 webserver.cc:460] Webserver started at http://127.0.103.65:42569/ using document root <none> and password file <none>
I20240430 17:02:41.791044 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:41.791292 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:41.791570 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:41.793015 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-0-root/instance:
uuid: "7b21e805c9134ce7842f2ef519c98616"
format_stamp: "Formatted at 2024-04-30 17:02:41 on dist-test-slave-x21r"
I20240430 17:02:41.795444 1714 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:41.796188 1714 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:41.798563 413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240430 17:02:41.802114 1755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:41.802901 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240430 17:02:41.803210 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-0-root
uuid: "7b21e805c9134ce7842f2ef519c98616"
format_stamp: "Formatted at 2024-04-30 17:02:41 on dist-test-slave-x21r"
I20240430 17:02:41.803480 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:41.819067 1733 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 17:02:41.821123 1733 catalog_manager.cc:1504] Loading token signing keys...
I20240430 17:02:41.824440 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:41.825773 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:41.827558 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:41.830133 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:41.830310 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:41.830559 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:41.830724 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:41.856012 1733 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa: Generated new TSK 0
I20240430 17:02:41.856765 1733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 17:02:41.891466 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:35247
I20240430 17:02:41.891558 1817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.65:35247 every 8 connection(s)
I20240430 17:02:41.903831 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240430 17:02:41.912606 1818 heartbeater.cc:347] Connected to a master server at 127.0.103.124:41743
I20240430 17:02:41.913148 1818 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:41.913096 1819 heartbeater.cc:347] Connected to a master server at 127.0.103.125:33953
I20240430 17:02:41.913888 1819 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:41.914280 1818 heartbeater.cc:511] Master 127.0.103.124:41743 requested a full tablet report, sending...
I20240430 17:02:41.914981 1819 heartbeater.cc:511] Master 127.0.103.125:33953 requested a full tablet report, sending...
I20240430 17:02:41.916954 1657 ts_manager.cc:194] Registered new tserver with Master: 7b21e805c9134ce7842f2ef519c98616 (127.0.103.65:35247)
I20240430 17:02:41.919004 1592 ts_manager.cc:194] Registered new tserver with Master: 7b21e805c9134ce7842f2ef519c98616 (127.0.103.65:35247)
I20240430 17:02:41.919461 1657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34584
W20240430 17:02:41.923560 1825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:41.924139 1820 heartbeater.cc:347] Connected to a master server at 127.0.103.126:36315
I20240430 17:02:41.924544 1820 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:41.925235 1820 heartbeater.cc:511] Master 127.0.103.126:36315 requested a full tablet report, sending...
W20240430 17:02:41.925350 1826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:41.927280 1828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:41.928092 1527 ts_manager.cc:194] Registered new tserver with Master: 7b21e805c9134ce7842f2ef519c98616 (127.0.103.65:35247)
I20240430 17:02:41.928625 413 server_base.cc:1034] running on GCE node
I20240430 17:02:41.929478 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:41.929682 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:41.929845 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496561929829 us; error 0 us; skew 500 ppm
I20240430 17:02:41.930316 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:41.932585 413 webserver.cc:460] Webserver started at http://127.0.103.66:38799/ using document root <none> and password file <none>
I20240430 17:02:41.933066 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:41.933252 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:41.933485 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:41.934463 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-1-root/instance:
uuid: "a9167663e46948d6ae96392d0a110e54"
format_stamp: "Formatted at 2024-04-30 17:02:41 on dist-test-slave-x21r"
I20240430 17:02:41.938642 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240430 17:02:41.941709 1833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:41.942420 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240430 17:02:41.942682 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-1-root
uuid: "a9167663e46948d6ae96392d0a110e54"
format_stamp: "Formatted at 2024-04-30 17:02:41 on dist-test-slave-x21r"
I20240430 17:02:41.942934 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:41.953725 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:41.954782 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:41.956136 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:41.958317 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:41.958488 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:41.958716 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:41.958853 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:41.997452 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:41293
I20240430 17:02:41.997524 1895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.66:41293 every 8 connection(s)
I20240430 17:02:42.014746 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240430 17:02:42.017503 1896 heartbeater.cc:347] Connected to a master server at 127.0.103.124:41743
I20240430 17:02:42.017947 1896 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:42.018555 1897 heartbeater.cc:347] Connected to a master server at 127.0.103.125:33953
I20240430 17:02:42.018818 1896 heartbeater.cc:511] Master 127.0.103.124:41743 requested a full tablet report, sending...
I20240430 17:02:42.018968 1897 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:42.019898 1897 heartbeater.cc:511] Master 127.0.103.125:33953 requested a full tablet report, sending...
I20240430 17:02:42.021627 1657 ts_manager.cc:194] Registered new tserver with Master: a9167663e46948d6ae96392d0a110e54 (127.0.103.66:41293)
I20240430 17:02:42.022735 1592 ts_manager.cc:194] Registered new tserver with Master: a9167663e46948d6ae96392d0a110e54 (127.0.103.66:41293)
I20240430 17:02:42.024011 1657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34596
I20240430 17:02:42.025652 1898 heartbeater.cc:347] Connected to a master server at 127.0.103.126:36315
I20240430 17:02:42.026049 1898 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:42.026798 1898 heartbeater.cc:511] Master 127.0.103.126:36315 requested a full tablet report, sending...
I20240430 17:02:42.029776 1527 ts_manager.cc:194] Registered new tserver with Master: a9167663e46948d6ae96392d0a110e54 (127.0.103.66:41293)
W20240430 17:02:42.030460 1903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:42.032088 1904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:42.032780 413 server_base.cc:1034] running on GCE node
W20240430 17:02:42.033565 1906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:42.034355 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:42.034583 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:42.034726 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496562034715 us; error 0 us; skew 500 ppm
I20240430 17:02:42.035282 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:42.037573 413 webserver.cc:460] Webserver started at http://127.0.103.67:33669/ using document root <none> and password file <none>
I20240430 17:02:42.037977 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:42.038136 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:42.038336 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:42.039347 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-2-root/instance:
uuid: "11625a022ac348f89e1732c8ce6429f5"
format_stamp: "Formatted at 2024-04-30 17:02:42 on dist-test-slave-x21r"
I20240430 17:02:42.043674 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240430 17:02:42.046784 1911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:42.047547 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240430 17:02:42.047879 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-2-root
uuid: "11625a022ac348f89e1732c8ce6429f5"
format_stamp: "Formatted at 2024-04-30 17:02:42 on dist-test-slave-x21r"
I20240430 17:02:42.048159 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714496547126677-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:42.056433 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:42.057507 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:42.058665 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:42.060781 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:42.060974 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:42.061209 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:42.061359 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:42.100214 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:41079
I20240430 17:02:42.100315 1973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.67:41079 every 8 connection(s)
I20240430 17:02:42.119175 1974 heartbeater.cc:347] Connected to a master server at 127.0.103.124:41743
I20240430 17:02:42.119678 1974 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:42.120527 1974 heartbeater.cc:511] Master 127.0.103.124:41743 requested a full tablet report, sending...
I20240430 17:02:42.122601 1657 ts_manager.cc:194] Registered new tserver with Master: 11625a022ac348f89e1732c8ce6429f5 (127.0.103.67:41079)
I20240430 17:02:42.124493 1657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34602
I20240430 17:02:42.126770 1975 heartbeater.cc:347] Connected to a master server at 127.0.103.125:33953
I20240430 17:02:42.127241 1975 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:42.127810 1976 heartbeater.cc:347] Connected to a master server at 127.0.103.126:36315
I20240430 17:02:42.127828 1975 heartbeater.cc:511] Master 127.0.103.125:33953 requested a full tablet report, sending...
I20240430 17:02:42.128297 1976 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:42.128855 1976 heartbeater.cc:511] Master 127.0.103.126:36315 requested a full tablet report, sending...
I20240430 17:02:42.130301 1592 ts_manager.cc:194] Registered new tserver with Master: 11625a022ac348f89e1732c8ce6429f5 (127.0.103.67:41079)
I20240430 17:02:42.131230 1527 ts_manager.cc:194] Registered new tserver with Master: 11625a022ac348f89e1732c8ce6429f5 (127.0.103.67:41079)
I20240430 17:02:42.132109 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018814674s
I20240430 17:02:42.665270 1708 catalog_manager.cc:1259] Loaded cluster ID: c2d7a0ed4a774a6887ae7d288fbf8152
I20240430 17:02:42.665570 1708 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81: loading cluster ID for follower catalog manager: success
I20240430 17:02:42.669644 1708 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81: acquiring CA information for follower catalog manager: success
I20240430 17:02:42.672678 1708 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240430 17:02:42.755818 1738 catalog_manager.cc:1259] Loaded cluster ID: c2d7a0ed4a774a6887ae7d288fbf8152
I20240430 17:02:42.756089 1738 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24: loading cluster ID for follower catalog manager: success
I20240430 17:02:42.760195 1738 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24: acquiring CA information for follower catalog manager: success
I20240430 17:02:42.763203 1738 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240430 17:02:42.922571 1818 heartbeater.cc:503] Master 127.0.103.124:41743 was elected leader, sending a full tablet report...
I20240430 17:02:43.027695 1896 heartbeater.cc:503] Master 127.0.103.124:41743 was elected leader, sending a full tablet report...
I20240430 17:02:43.126947 1974 heartbeater.cc:503] Master 127.0.103.124:41743 was elected leader, sending a full tablet report...
I20240430 17:02:43.724745 1728 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:02:44.209318 413 test_util.cc:268] Using random seed: 1454220943
I20240430 17:02:44.246642 1657 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34614:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 17:02:44.316470 1861 tablet_service.cc:1467] Processing CreateTablet for tablet aef6154aa3454b7dbdfcb2f6d9056502 (DEFAULT_TABLE table=test-workload [id=fa2358079c6d4a568ecf89bc2887c814]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240430 17:02:44.317834 1861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef6154aa3454b7dbdfcb2f6d9056502. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:44.317831 1939 tablet_service.cc:1467] Processing CreateTablet for tablet 6f03ef6ee9ff461198f2c01843b5a77b (DEFAULT_TABLE table=test-workload [id=fa2358079c6d4a568ecf89bc2887c814]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240430 17:02:44.319181 1939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f03ef6ee9ff461198f2c01843b5a77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:44.326277 1783 tablet_service.cc:1467] Processing CreateTablet for tablet a6fd54b6a745464eb998ed8229fb6ce4 (DEFAULT_TABLE table=test-workload [id=fa2358079c6d4a568ecf89bc2887c814]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240430 17:02:44.327486 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6fd54b6a745464eb998ed8229fb6ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:44.331401 2010 tablet_bootstrap.cc:492] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54: Bootstrap starting.
I20240430 17:02:44.336421 2010 tablet_bootstrap.cc:654] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:44.340585 2012 tablet_bootstrap.cc:492] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5: Bootstrap starting.
I20240430 17:02:44.343092 2010 tablet_bootstrap.cc:492] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54: No bootstrap required, opened a new log
I20240430 17:02:44.343819 2010 ts_tablet_manager.cc:1367] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240430 17:02:44.344409 2013 tablet_bootstrap.cc:492] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616: Bootstrap starting.
I20240430 17:02:44.345407 2012 tablet_bootstrap.cc:654] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:44.346153 2010 raft_consensus.cc:348] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9167663e46948d6ae96392d0a110e54" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41293 } }
I20240430 17:02:44.346608 2010 raft_consensus.cc:374] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:44.346830 2010 raft_consensus.cc:729] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9167663e46948d6ae96392d0a110e54, State: Initialized, Role: FOLLOWER
I20240430 17:02:44.347402 2010 consensus_queue.cc:260] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9167663e46948d6ae96392d0a110e54" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41293 } }
I20240430 17:02:44.348016 2010 raft_consensus.cc:388] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:02:44.348330 2010 raft_consensus.cc:482] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:02:44.348675 2010 raft_consensus.cc:3037] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:44.352847 2013 tablet_bootstrap.cc:654] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:44.353672 2010 raft_consensus.cc:504] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9167663e46948d6ae96392d0a110e54" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41293 } }
I20240430 17:02:44.354249 2010 leader_election.cc:302] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9167663e46948d6ae96392d0a110e54; no voters:
I20240430 17:02:44.358654 2010 leader_election.cc:288] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:02:44.358924 2012 tablet_bootstrap.cc:492] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5: No bootstrap required, opened a new log
I20240430 17:02:44.359587 2012 ts_tablet_manager.cc:1367] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.010s
I20240430 17:02:44.359063 2015 raft_consensus.cc:2781] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:44.362771 2010 ts_tablet_manager.cc:1398] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54: Time spent starting tablet: real 0.019s user 0.012s sys 0.004s
I20240430 17:02:44.364184 2012 raft_consensus.cc:348] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11625a022ac348f89e1732c8ce6429f5" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41079 } }
I20240430 17:02:44.364849 2012 raft_consensus.cc:374] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:44.365144 2012 raft_consensus.cc:729] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11625a022ac348f89e1732c8ce6429f5, State: Initialized, Role: FOLLOWER
I20240430 17:02:44.365897 2012 consensus_queue.cc:260] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11625a022ac348f89e1732c8ce6429f5" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41079 } }
I20240430 17:02:44.366562 2012 raft_consensus.cc:388] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:02:44.366883 2012 raft_consensus.cc:482] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:02:44.367228 2012 raft_consensus.cc:3037] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:44.369215 2013 tablet_bootstrap.cc:492] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616: No bootstrap required, opened a new log
I20240430 17:02:44.369693 2013 ts_tablet_manager.cc:1367] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.004s
I20240430 17:02:44.372352 2013 raft_consensus.cc:348] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b21e805c9134ce7842f2ef519c98616" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35247 } }
I20240430 17:02:44.372882 2013 raft_consensus.cc:374] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:44.373147 2013 raft_consensus.cc:729] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b21e805c9134ce7842f2ef519c98616, State: Initialized, Role: FOLLOWER
I20240430 17:02:44.373783 2013 consensus_queue.cc:260] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b21e805c9134ce7842f2ef519c98616" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35247 } }
I20240430 17:02:44.374413 2013 raft_consensus.cc:388] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:02:44.374693 2012 raft_consensus.cc:504] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11625a022ac348f89e1732c8ce6429f5" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41079 } }
I20240430 17:02:44.374709 2013 raft_consensus.cc:482] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:02:44.375408 2015 raft_consensus.cc:686] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [term 1 LEADER]: Becoming Leader. State: Replica: a9167663e46948d6ae96392d0a110e54, State: Running, Role: LEADER
I20240430 17:02:44.375501 2013 raft_consensus.cc:3037] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:44.375586 2012 leader_election.cc:302] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11625a022ac348f89e1732c8ce6429f5; no voters:
I20240430 17:02:44.376186 2015 consensus_queue.cc:237] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9167663e46948d6ae96392d0a110e54" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41293 } }
I20240430 17:02:44.382735 2013 raft_consensus.cc:504] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b21e805c9134ce7842f2ef519c98616" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35247 } }
I20240430 17:02:44.383496 2013 leader_election.cc:302] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b21e805c9134ce7842f2ef519c98616; no voters:
I20240430 17:02:44.397195 1657 catalog_manager.cc:5484] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9167663e46948d6ae96392d0a110e54 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a9167663e46948d6ae96392d0a110e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9167663e46948d6ae96392d0a110e54" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41293 } health_report { overall_health: HEALTHY } } }
I20240430 17:02:44.403962 2012 leader_election.cc:288] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:02:44.405601 2019 raft_consensus.cc:2781] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:44.427496 2012 ts_tablet_manager.cc:1398] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5: Time spent starting tablet: real 0.067s user 0.023s sys 0.006s
I20240430 17:02:44.428469 2019 raft_consensus.cc:686] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [term 1 LEADER]: Becoming Leader. State: Replica: 11625a022ac348f89e1732c8ce6429f5, State: Running, Role: LEADER
I20240430 17:02:44.429518 2019 consensus_queue.cc:237] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11625a022ac348f89e1732c8ce6429f5" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41079 } }
I20240430 17:02:44.445279 2022 raft_consensus.cc:2781] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:44.443526 2013 leader_election.cc:288] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:02:44.449203 2022 raft_consensus.cc:686] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [term 1 LEADER]: Becoming Leader. State: Replica: 7b21e805c9134ce7842f2ef519c98616, State: Running, Role: LEADER
I20240430 17:02:44.449918 2022 consensus_queue.cc:237] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b21e805c9134ce7842f2ef519c98616" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35247 } }
I20240430 17:02:44.457473 1656 catalog_manager.cc:5484] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11625a022ac348f89e1732c8ce6429f5 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "11625a022ac348f89e1732c8ce6429f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11625a022ac348f89e1732c8ce6429f5" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41079 } health_report { overall_health: HEALTHY } } }
I20240430 17:02:44.468209 2013 ts_tablet_manager.cc:1398] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616: Time spent starting tablet: real 0.098s user 0.052s sys 0.022s
I20240430 17:02:44.471364 1657 catalog_manager.cc:5484] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b21e805c9134ce7842f2ef519c98616 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7b21e805c9134ce7842f2ef519c98616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b21e805c9134ce7842f2ef519c98616" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35247 } health_report { overall_health: HEALTHY } } }
I20240430 17:02:44.516558 413 master.cc:537] Master@127.0.103.124:41743 shutting down...
I20240430 17:02:44.531774 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:44.532572 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:44.532912 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P e1ef14b5433f4dbc885454e5b7eb79fa: stopping tablet replica
I20240430 17:02:44.552738 413 master.cc:552] Master@127.0.103.124:41743 shutdown complete.
W20240430 17:02:45.455018 1896 heartbeater.cc:644] Failed to heartbeat to 127.0.103.124:41743 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.103.124:41743: connect: Connection refused (error 111)
I20240430 17:02:46.010843 2034 raft_consensus.cc:482] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1ef14b5433f4dbc885454e5b7eb79fa)
I20240430 17:02:46.011399 2034 raft_consensus.cc:504] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:46.012027 2035 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1ef14b5433f4dbc885454e5b7eb79fa)
I20240430 17:02:46.012425 2035 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:46.012972 2034 leader_election.cc:288] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca9980152ed94b4395d601f62e7b8c81 (127.0.103.126:36315), e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743)
I20240430 17:02:46.013820 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a02319a009aa481584f555e8b3d77b24" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ca9980152ed94b4395d601f62e7b8c81" is_pre_election: true
I20240430 17:02:46.014281 2035 leader_election.cc:288] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a02319a009aa481584f555e8b3d77b24 (127.0.103.125:33953), e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743)
I20240430 17:02:46.014446 1537 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a02319a009aa481584f555e8b3d77b24 in term 1.
I20240430 17:02:46.015676 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca9980152ed94b4395d601f62e7b8c81" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "a02319a009aa481584f555e8b3d77b24" is_pre_election: true
I20240430 17:02:46.015823 1576 leader_election.cc:302] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a02319a009aa481584f555e8b3d77b24, ca9980152ed94b4395d601f62e7b8c81; no voters:
I20240430 17:02:46.016567 1602 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca9980152ed94b4395d601f62e7b8c81 in term 1.
W20240430 17:02:46.016860 1576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.124:41743: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20240430 17:02:46.017096 2034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 17:02:46.017505 2034 raft_consensus.cc:482] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1ef14b5433f4dbc885454e5b7eb79fa)
I20240430 17:02:46.017879 2034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 1 FOLLOWER]: Advancing to term 2
I20240430 17:02:46.017798 1513 leader_election.cc:302] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a02319a009aa481584f555e8b3d77b24, ca9980152ed94b4395d601f62e7b8c81; no voters:
I20240430 17:02:46.018669 2035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 17:02:46.019027 2035 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1ef14b5433f4dbc885454e5b7eb79fa)
I20240430 17:02:46.019397 2035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 1 FOLLOWER]: Advancing to term 2
I20240430 17:02:46.024744 2034 raft_consensus.cc:504] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:46.026993 2034 leader_election.cc:288] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [CANDIDATE]: Term 2 election: Requested vote from peers ca9980152ed94b4395d601f62e7b8c81 (127.0.103.126:36315), e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743)
I20240430 17:02:46.027805 2035 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:46.029825 2035 leader_election.cc:288] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [CANDIDATE]: Term 2 election: Requested vote from peers a02319a009aa481584f555e8b3d77b24 (127.0.103.125:33953), e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743)
I20240430 17:02:46.030052 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca9980152ed94b4395d601f62e7b8c81" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "a02319a009aa481584f555e8b3d77b24"
I20240430 17:02:46.030833 1602 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca9980152ed94b4395d601f62e7b8c81 in current term 2: Already voted for candidate a02319a009aa481584f555e8b3d77b24 in this term.
W20240430 17:02:46.030731 1511 leader_election.cc:334] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743): Network error: Client connection negotiation failed: client connection to 127.0.103.124:41743: connect: Connection refused (error 111)
I20240430 17:02:46.031251 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a02319a009aa481584f555e8b3d77b24" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ca9980152ed94b4395d601f62e7b8c81"
I20240430 17:02:46.032004 1537 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a02319a009aa481584f555e8b3d77b24 in current term 2: Already voted for candidate ca9980152ed94b4395d601f62e7b8c81 in this term.
W20240430 17:02:46.032702 1576 leader_election.cc:334] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743): Network error: Client connection negotiation failed: client connection to 127.0.103.124:41743: connect: Connection refused (error 111)
W20240430 17:02:46.033867 1511 leader_election.cc:334] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743): Network error: Client connection negotiation failed: client connection to 127.0.103.124:41743: connect: Connection refused (error 111)
I20240430 17:02:46.034278 1511 leader_election.cc:302] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca9980152ed94b4395d601f62e7b8c81; no voters: a02319a009aa481584f555e8b3d77b24, e1ef14b5433f4dbc885454e5b7eb79fa
I20240430 17:02:46.034845 2035 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240430 17:02:46.035223 1576 leader_election.cc:334] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743): Network error: Client connection negotiation failed: client connection to 127.0.103.124:41743: connect: Connection refused (error 111)
I20240430 17:02:46.035539 1576 leader_election.cc:302] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a02319a009aa481584f555e8b3d77b24; no voters: ca9980152ed94b4395d601f62e7b8c81, e1ef14b5433f4dbc885454e5b7eb79fa
I20240430 17:02:46.036034 2034 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240430 17:02:47.727550 2047 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:47.728219 2047 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:47.729997 2047 leader_election.cc:288] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a02319a009aa481584f555e8b3d77b24 (127.0.103.125:33953), e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743)
I20240430 17:02:47.731648 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca9980152ed94b4395d601f62e7b8c81" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "a02319a009aa481584f555e8b3d77b24" is_pre_election: true
I20240430 17:02:47.732497 1602 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca9980152ed94b4395d601f62e7b8c81 in term 2.
I20240430 17:02:47.733830 1513 leader_election.cc:302] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a02319a009aa481584f555e8b3d77b24, ca9980152ed94b4395d601f62e7b8c81; no voters:
I20240430 17:02:47.734616 2047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240430 17:02:47.734994 2047 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:47.735261 2047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 2 FOLLOWER]: Advancing to term 3
W20240430 17:02:47.736711 1511 leader_election.cc:334] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743): Network error: Client connection negotiation failed: client connection to 127.0.103.124:41743: connect: Connection refused (error 111)
I20240430 17:02:47.741691 2047 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:47.743420 2047 leader_election.cc:288] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [CANDIDATE]: Term 3 election: Requested vote from peers a02319a009aa481584f555e8b3d77b24 (127.0.103.125:33953), e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743)
I20240430 17:02:47.744733 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca9980152ed94b4395d601f62e7b8c81" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "a02319a009aa481584f555e8b3d77b24"
I20240430 17:02:47.745365 1602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 2 FOLLOWER]: Advancing to term 3
W20240430 17:02:47.747398 1511 leader_election.cc:334] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743): Network error: Client connection negotiation failed: client connection to 127.0.103.124:41743: connect: Connection refused (error 111)
I20240430 17:02:47.751142 1602 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9980152ed94b4395d601f62e7b8c81 in term 3.
I20240430 17:02:47.752135 1513 leader_election.cc:302] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a02319a009aa481584f555e8b3d77b24, ca9980152ed94b4395d601f62e7b8c81; no voters: e1ef14b5433f4dbc885454e5b7eb79fa
I20240430 17:02:47.753163 2047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 3 FOLLOWER]: Leader election won for term 3
I20240430 17:02:47.754269 2047 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 3 LEADER]: Becoming Leader. State: Replica: ca9980152ed94b4395d601f62e7b8c81, State: Running, Role: LEADER
I20240430 17:02:47.754966 2047 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } }
I20240430 17:02:47.761210 2050 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca9980152ed94b4395d601f62e7b8c81. Latest consensus state: current_term: 3 leader_uuid: "ca9980152ed94b4395d601f62e7b8c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:47.761788 2050 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [sys.catalog]: This master's current role is: LEADER
I20240430 17:02:47.763221 2052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 17:02:47.768746 2052 catalog_manager.cc:669] Loaded metadata for table test-workload [id=fa2358079c6d4a568ecf89bc2887c814]
I20240430 17:02:47.773273 2052 tablet_loader.cc:96] loaded metadata for tablet 6f03ef6ee9ff461198f2c01843b5a77b (table test-workload [id=fa2358079c6d4a568ecf89bc2887c814])
I20240430 17:02:47.774204 2052 tablet_loader.cc:96] loaded metadata for tablet a6fd54b6a745464eb998ed8229fb6ce4 (table test-workload [id=fa2358079c6d4a568ecf89bc2887c814])
I20240430 17:02:47.775090 2052 tablet_loader.cc:96] loaded metadata for tablet aef6154aa3454b7dbdfcb2f6d9056502 (table test-workload [id=fa2358079c6d4a568ecf89bc2887c814])
I20240430 17:02:47.775957 2052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 17:02:47.778793 2052 catalog_manager.cc:1259] Loaded cluster ID: c2d7a0ed4a774a6887ae7d288fbf8152
I20240430 17:02:47.779003 2052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 17:02:47.782271 2052 catalog_manager.cc:1504] Loading token signing keys...
I20240430 17:02:47.785130 2052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81: Loaded TSK: 0
I20240430 17:02:47.786020 2052 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240430 17:02:48.171063 1511 consensus_peers.cc:485] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 -> Peer e1ef14b5433f4dbc885454e5b7eb79fa (127.0.103.124:41743): Couldn't send request to peer e1ef14b5433f4dbc885454e5b7eb79fa. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.124:41743: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 17:02:48.334158 1602 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 3 FOLLOWER]: Refusing update from remote peer ca9980152ed94b4395d601f62e7b8c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240430 17:02:48.335568 2050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240430 17:02:48.340591 2053 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca9980152ed94b4395d601f62e7b8c81. Latest consensus state: current_term: 3 leader_uuid: "ca9980152ed94b4395d601f62e7b8c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:48.341260 2053 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:48.344200 2055 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ca9980152ed94b4395d601f62e7b8c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:48.344930 2055 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [sys.catalog]: This master's current role is: LEADER
I20240430 17:02:48.345942 2053 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ca9980152ed94b4395d601f62e7b8c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9980152ed94b4395d601f62e7b8c81" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 36315 } } peers { permanent_uuid: "a02319a009aa481584f555e8b3d77b24" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 33953 } } peers { permanent_uuid: "e1ef14b5433f4dbc885454e5b7eb79fa" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41743 } } }
I20240430 17:02:48.346719 2053 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 17:02:49.464973 1898 heartbeater.cc:503] Master 127.0.103.126:36315 was elected leader, sending a full tablet report...
I20240430 17:02:49.494608 1820 heartbeater.cc:503] Master 127.0.103.126:36315 was elected leader, sending a full tablet report...
I20240430 17:02:49.513149 1976 heartbeater.cc:503] Master 127.0.103.126:36315 was elected leader, sending a full tablet report...
I20240430 17:02:49.658928 1707 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:02:49.659462 1707 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:02:49.829329 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20240430 17:02:49.857088 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:02:49.857681 413 tablet_replica.cc:330] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616: stopping tablet replica
I20240430 17:02:49.858158 413 raft_consensus.cc:2229] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:49.858592 413 raft_consensus.cc:2258] T a6fd54b6a745464eb998ed8229fb6ce4 P 7b21e805c9134ce7842f2ef519c98616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:49.877480 413 tablet_server.cc:194] TabletServer@127.0.103.65:0 shutdown complete.
I20240430 17:02:49.888374 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20240430 17:02:49.912451 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:02:49.913230 413 tablet_replica.cc:330] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54: stopping tablet replica
I20240430 17:02:49.913830 413 raft_consensus.cc:2229] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:49.914703 413 raft_consensus.cc:2258] T aef6154aa3454b7dbdfcb2f6d9056502 P a9167663e46948d6ae96392d0a110e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:49.933158 413 tablet_server.cc:194] TabletServer@127.0.103.66:0 shutdown complete.
I20240430 17:02:49.942953 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20240430 17:02:49.965909 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:02:49.966545 413 tablet_replica.cc:330] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5: stopping tablet replica
I20240430 17:02:49.967056 413 raft_consensus.cc:2229] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:02:49.967532 413 raft_consensus.cc:2258] T 6f03ef6ee9ff461198f2c01843b5a77b P 11625a022ac348f89e1732c8ce6429f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:49.985633 413 tablet_server.cc:194] TabletServer@127.0.103.67:0 shutdown complete.
I20240430 17:02:49.994766 413 master.cc:537] Master@127.0.103.126:36315 shutting down...
I20240430 17:02:50.012363 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 3 LEADER]: Raft consensus shutting down.
I20240430 17:02:50.013199 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:50.013516 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P ca9980152ed94b4395d601f62e7b8c81: stopping tablet replica
I20240430 17:02:50.032060 413 master.cc:552] Master@127.0.103.126:36315 shutdown complete.
I20240430 17:02:50.046094 413 master.cc:537] Master@127.0.103.125:33953 shutting down...
I20240430 17:02:50.061884 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240430 17:02:50.062340 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240430 17:02:50.062605 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P a02319a009aa481584f555e8b3d77b24: stopping tablet replica
I20240430 17:02:50.080130 413 master.cc:552] Master@127.0.103.125:33953 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (8799 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240430 17:02:50.109829 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:39231
I20240430 17:02:50.110864 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:50.117365 2067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:50.118216 2068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:50.119046 2070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:50.119758 413 server_base.cc:1034] running on GCE node
I20240430 17:02:50.120465 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:50.120636 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:50.120788 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496570120764 us; error 0 us; skew 500 ppm
I20240430 17:02:50.121237 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:50.199139 413 webserver.cc:460] Webserver started at http://127.0.103.126:39363/ using document root <none> and password file <none>
I20240430 17:02:50.199663 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:50.199841 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:50.200145 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:50.201429 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/master-0-root/instance:
uuid: "f5afb6fc1dc24fcd9bdbe41d8e27c965"
format_stamp: "Formatted at 2024-04-30 17:02:50 on dist-test-slave-x21r"
I20240430 17:02:50.205749 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240430 17:02:50.208962 2075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:50.209708 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240430 17:02:50.209975 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/master-0-root
uuid: "f5afb6fc1dc24fcd9bdbe41d8e27c965"
format_stamp: "Formatted at 2024-04-30 17:02:50 on dist-test-slave-x21r"
I20240430 17:02:50.210227 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:50.238127 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:50.239385 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:50.275221 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:39231
I20240430 17:02:50.275363 2126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.126:39231 every 8 connection(s)
I20240430 17:02:50.279067 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:50.289011 2127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965: Bootstrap starting.
I20240430 17:02:50.293020 2127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:50.296918 2127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965: No bootstrap required, opened a new log
I20240430 17:02:50.298664 2127 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5afb6fc1dc24fcd9bdbe41d8e27c965" member_type: VOTER }
I20240430 17:02:50.299018 2127 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:50.299259 2127 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5afb6fc1dc24fcd9bdbe41d8e27c965, State: Initialized, Role: FOLLOWER
I20240430 17:02:50.299783 2127 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5afb6fc1dc24fcd9bdbe41d8e27c965" member_type: VOTER }
I20240430 17:02:50.300194 2127 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:02:50.300383 2127 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:02:50.300583 2127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:50.304773 2127 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5afb6fc1dc24fcd9bdbe41d8e27c965" member_type: VOTER }
I20240430 17:02:50.305212 2127 leader_election.cc:302] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5afb6fc1dc24fcd9bdbe41d8e27c965; no voters:
I20240430 17:02:50.306210 2127 leader_election.cc:288] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:02:50.306594 2130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:50.307896 2130 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [term 1 LEADER]: Becoming Leader. State: Replica: f5afb6fc1dc24fcd9bdbe41d8e27c965, State: Running, Role: LEADER
I20240430 17:02:50.308446 2130 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5afb6fc1dc24fcd9bdbe41d8e27c965" member_type: VOTER }
I20240430 17:02:50.309095 2127 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 17:02:50.311322 2131 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5afb6fc1dc24fcd9bdbe41d8e27c965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5afb6fc1dc24fcd9bdbe41d8e27c965" member_type: VOTER } }
I20240430 17:02:50.311272 2132 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5afb6fc1dc24fcd9bdbe41d8e27c965. Latest consensus state: current_term: 1 leader_uuid: "f5afb6fc1dc24fcd9bdbe41d8e27c965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5afb6fc1dc24fcd9bdbe41d8e27c965" member_type: VOTER } }
I20240430 17:02:50.312052 2131 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [sys.catalog]: This master's current role is: LEADER
I20240430 17:02:50.312119 2132 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [sys.catalog]: This master's current role is: LEADER
I20240430 17:02:50.316143 2135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 17:02:50.323752 2135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 17:02:50.324661 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240430 17:02:50.331890 2135 catalog_manager.cc:1347] Generated new cluster ID: 31f66fdbe08b474f91c554465985d8fc
I20240430 17:02:50.332165 2135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 17:02:50.348578 2135 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 17:02:50.349783 2135 catalog_manager.cc:1504] Loading token signing keys...
I20240430 17:02:50.362813 2135 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965: Generated new TSK 0
I20240430 17:02:50.363427 2135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 17:02:50.391424 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:50.397738 2148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:50.398629 2149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:50.399900 413 server_base.cc:1034] running on GCE node
W20240430 17:02:50.400668 2151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:50.401407 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:50.401609 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:50.401774 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496570401755 us; error 0 us; skew 500 ppm
I20240430 17:02:50.402309 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:50.404570 413 webserver.cc:460] Webserver started at http://127.0.103.65:41121/ using document root <none> and password file <none>
I20240430 17:02:50.404995 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:50.405172 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:50.405412 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:50.406455 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-0-root/instance:
uuid: "be688569c19940fe856c07b5d80f7202"
format_stamp: "Formatted at 2024-04-30 17:02:50 on dist-test-slave-x21r"
I20240430 17:02:50.410974 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240430 17:02:50.414036 2156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:50.414733 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240430 17:02:50.414973 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-0-root
uuid: "be688569c19940fe856c07b5d80f7202"
format_stamp: "Formatted at 2024-04-30 17:02:50 on dist-test-slave-x21r"
I20240430 17:02:50.415263 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:50.434423 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:50.435546 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:50.436915 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:50.439164 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:50.439412 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:50.439663 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:50.439869 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:50.477659 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:43615
I20240430 17:02:50.477756 2218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.65:43615 every 8 connection(s)
I20240430 17:02:50.482048 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:50.489789 2223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:50.490533 2224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:50.492192 2226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:50.493124 413 server_base.cc:1034] running on GCE node
I20240430 17:02:50.494131 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:50.494392 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:50.494562 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496570494544 us; error 0 us; skew 500 ppm
I20240430 17:02:50.495146 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:50.495241 2219 heartbeater.cc:347] Connected to a master server at 127.0.103.126:39231
I20240430 17:02:50.495628 2219 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:50.496486 2219 heartbeater.cc:511] Master 127.0.103.126:39231 requested a full tablet report, sending...
I20240430 17:02:50.497792 413 webserver.cc:460] Webserver started at http://127.0.103.66:33293/ using document root <none> and password file <none>
I20240430 17:02:50.498260 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:50.498454 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:50.498533 2092 ts_manager.cc:194] Registered new tserver with Master: be688569c19940fe856c07b5d80f7202 (127.0.103.65:43615)
I20240430 17:02:50.498700 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:50.499872 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-1-root/instance:
uuid: "7ba0946bd92b47a6a06a1d09de274056"
format_stamp: "Formatted at 2024-04-30 17:02:50 on dist-test-slave-x21r"
I20240430 17:02:50.500443 2092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57588
I20240430 17:02:50.504269 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240430 17:02:50.507365 2231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:50.508096 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240430 17:02:50.508359 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-1-root
uuid: "7ba0946bd92b47a6a06a1d09de274056"
format_stamp: "Formatted at 2024-04-30 17:02:50 on dist-test-slave-x21r"
I20240430 17:02:50.508620 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:50.517398 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:50.518491 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:50.519788 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:50.521864 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:50.522055 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:50.522264 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:50.522415 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:50.558758 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:46531
I20240430 17:02:50.558852 2293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.66:46531 every 8 connection(s)
I20240430 17:02:50.563249 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:02:50.570896 2297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:02:50.573812 2298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:50.575366 2294 heartbeater.cc:347] Connected to a master server at 127.0.103.126:39231
I20240430 17:02:50.575788 2294 heartbeater.cc:464] Registering TS with master...
W20240430 17:02:50.575693 2300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:02:50.576372 413 server_base.cc:1034] running on GCE node
I20240430 17:02:50.576714 2294 heartbeater.cc:511] Master 127.0.103.126:39231 requested a full tablet report, sending...
I20240430 17:02:50.577199 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:02:50.577415 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:02:50.577618 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496570577600 us; error 0 us; skew 500 ppm
I20240430 17:02:50.578203 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:02:50.578539 2092 ts_manager.cc:194] Registered new tserver with Master: 7ba0946bd92b47a6a06a1d09de274056 (127.0.103.66:46531)
I20240430 17:02:50.580089 2092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57602
I20240430 17:02:50.580866 413 webserver.cc:460] Webserver started at http://127.0.103.67:37161/ using document root <none> and password file <none>
I20240430 17:02:50.581411 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:02:50.581594 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:02:50.581838 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:02:50.582947 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-2-root/instance:
uuid: "ba35a6d0a71d46909505e61bb9884415"
format_stamp: "Formatted at 2024-04-30 17:02:50 on dist-test-slave-x21r"
I20240430 17:02:50.587020 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240430 17:02:50.589939 2305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:50.590646 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240430 17:02:50.590902 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-2-root
uuid: "ba35a6d0a71d46909505e61bb9884415"
format_stamp: "Formatted at 2024-04-30 17:02:50 on dist-test-slave-x21r"
I20240430 17:02:50.591192 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:02:50.614434 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:02:50.615556 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:02:50.617030 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:02:50.619237 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:02:50.619434 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:50.619648 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:02:50.619796 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:02:50.656488 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:41179
I20240430 17:02:50.656582 2367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.67:41179 every 8 connection(s)
I20240430 17:02:50.669221 2368 heartbeater.cc:347] Connected to a master server at 127.0.103.126:39231
I20240430 17:02:50.669567 2368 heartbeater.cc:464] Registering TS with master...
I20240430 17:02:50.670255 2368 heartbeater.cc:511] Master 127.0.103.126:39231 requested a full tablet report, sending...
I20240430 17:02:50.671900 2092 ts_manager.cc:194] Registered new tserver with Master: ba35a6d0a71d46909505e61bb9884415 (127.0.103.67:41179)
I20240430 17:02:50.672894 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013350743s
I20240430 17:02:50.673105 2092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57612
I20240430 17:02:51.502749 2219 heartbeater.cc:503] Master 127.0.103.126:39231 was elected leader, sending a full tablet report...
I20240430 17:02:51.582286 2294 heartbeater.cc:503] Master 127.0.103.126:39231 was elected leader, sending a full tablet report...
I20240430 17:02:51.675455 2368 heartbeater.cc:503] Master 127.0.103.126:39231 was elected leader, sending a full tablet report...
I20240430 17:02:51.704792 413 test_util.cc:268] Using random seed: 1461716417
I20240430 17:02:51.726928 2092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57614:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 17:02:51.729048 2092 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 17:02:51.773715 2333 tablet_service.cc:1467] Processing CreateTablet for tablet 293b1c0771ea4a3cab5f279bdc069c72 (DEFAULT_TABLE table=test-workload [id=4a4b2d99900349fd8c6834d344a2cc3e]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240430 17:02:51.774127 2332 tablet_service.cc:1467] Processing CreateTablet for tablet 0d26eca405014074995129f798290254 (DEFAULT_TABLE table=test-workload [id=4a4b2d99900349fd8c6834d344a2cc3e]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240430 17:02:51.775192 2333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293b1c0771ea4a3cab5f279bdc069c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:51.776530 2332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d26eca405014074995129f798290254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:51.782182 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 0d26eca405014074995129f798290254 (DEFAULT_TABLE table=test-workload [id=4a4b2d99900349fd8c6834d344a2cc3e]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240430 17:02:51.782182 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 293b1c0771ea4a3cab5f279bdc069c72 (DEFAULT_TABLE table=test-workload [id=4a4b2d99900349fd8c6834d344a2cc3e]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240430 17:02:51.783740 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d26eca405014074995129f798290254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:51.783618 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 293b1c0771ea4a3cab5f279bdc069c72 (DEFAULT_TABLE table=test-workload [id=4a4b2d99900349fd8c6834d344a2cc3e]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240430 17:02:51.784494 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 0d26eca405014074995129f798290254 (DEFAULT_TABLE table=test-workload [id=4a4b2d99900349fd8c6834d344a2cc3e]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240430 17:02:51.784907 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293b1c0771ea4a3cab5f279bdc069c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:51.785709 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d26eca405014074995129f798290254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:51.786412 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293b1c0771ea4a3cab5f279bdc069c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:02:51.798976 2388 tablet_bootstrap.cc:492] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415: Bootstrap starting.
I20240430 17:02:51.807848 2388 tablet_bootstrap.cc:654] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:51.808141 2389 tablet_bootstrap.cc:492] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056: Bootstrap starting.
I20240430 17:02:51.819074 2390 tablet_bootstrap.cc:492] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202: Bootstrap starting.
I20240430 17:02:51.819665 2389 tablet_bootstrap.cc:654] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:51.822007 2388 tablet_bootstrap.cc:492] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415: No bootstrap required, opened a new log
I20240430 17:02:51.822517 2388 ts_tablet_manager.cc:1367] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.000s
I20240430 17:02:51.824492 2388 raft_consensus.cc:348] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } }
I20240430 17:02:51.825098 2388 raft_consensus.cc:374] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:51.825409 2388 raft_consensus.cc:729] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba35a6d0a71d46909505e61bb9884415, State: Initialized, Role: FOLLOWER
I20240430 17:02:51.826330 2390 tablet_bootstrap.cc:654] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:51.826222 2388 consensus_queue.cc:260] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } }
I20240430 17:02:51.829409 2388 ts_tablet_manager.cc:1398] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20240430 17:02:51.830360 2388 tablet_bootstrap.cc:492] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415: Bootstrap starting.
I20240430 17:02:51.830518 2389 tablet_bootstrap.cc:492] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056: No bootstrap required, opened a new log
I20240430 17:02:51.831162 2389 ts_tablet_manager.cc:1367] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.004s
I20240430 17:02:51.834502 2389 raft_consensus.cc:348] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } }
I20240430 17:02:51.835496 2389 raft_consensus.cc:374] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:51.835819 2388 tablet_bootstrap.cc:654] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:51.835805 2389 raft_consensus.cc:729] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba0946bd92b47a6a06a1d09de274056, State: Initialized, Role: FOLLOWER
I20240430 17:02:51.836956 2389 consensus_queue.cc:260] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } }
I20240430 17:02:51.838323 2390 tablet_bootstrap.cc:492] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202: No bootstrap required, opened a new log
I20240430 17:02:51.838985 2390 ts_tablet_manager.cc:1367] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20240430 17:02:51.842428 2390 raft_consensus.cc:348] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } }
I20240430 17:02:51.843617 2390 raft_consensus.cc:374] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:51.844035 2390 raft_consensus.cc:729] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be688569c19940fe856c07b5d80f7202, State: Initialized, Role: FOLLOWER
I20240430 17:02:51.844892 2390 consensus_queue.cc:260] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } }
I20240430 17:02:51.846562 2389 ts_tablet_manager.cc:1398] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056: Time spent starting tablet: real 0.015s user 0.006s sys 0.007s
I20240430 17:02:51.847633 2389 tablet_bootstrap.cc:492] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056: Bootstrap starting.
I20240430 17:02:51.851787 2388 tablet_bootstrap.cc:492] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415: No bootstrap required, opened a new log
I20240430 17:02:51.852340 2388 ts_tablet_manager.cc:1367] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.004s
I20240430 17:02:51.853130 2389 tablet_bootstrap.cc:654] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:51.853489 2390 ts_tablet_manager.cc:1398] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202: Time spent starting tablet: real 0.014s user 0.006s sys 0.003s
I20240430 17:02:51.854846 2388 raft_consensus.cc:348] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } } peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } }
I20240430 17:02:51.855676 2388 raft_consensus.cc:374] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:51.856011 2388 raft_consensus.cc:729] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba35a6d0a71d46909505e61bb9884415, State: Initialized, Role: FOLLOWER
I20240430 17:02:51.856123 2390 tablet_bootstrap.cc:492] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202: Bootstrap starting.
I20240430 17:02:51.856771 2388 consensus_queue.cc:260] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } } peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } }
I20240430 17:02:51.858973 2388 ts_tablet_manager.cc:1398] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240430 17:02:51.864816 2389 tablet_bootstrap.cc:492] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056: No bootstrap required, opened a new log
I20240430 17:02:51.865386 2389 ts_tablet_manager.cc:1367] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.001s
I20240430 17:02:51.865562 2390 tablet_bootstrap.cc:654] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202: Neither blocks nor log segments found. Creating new log.
I20240430 17:02:51.868134 2389 raft_consensus.cc:348] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } } peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } }
I20240430 17:02:51.869086 2389 raft_consensus.cc:374] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:51.869419 2389 raft_consensus.cc:729] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba0946bd92b47a6a06a1d09de274056, State: Initialized, Role: FOLLOWER
I20240430 17:02:51.870159 2389 consensus_queue.cc:260] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } } peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } }
I20240430 17:02:51.872251 2389 ts_tablet_manager.cc:1398] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240430 17:02:51.872291 2390 tablet_bootstrap.cc:492] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202: No bootstrap required, opened a new log
I20240430 17:02:51.872936 2390 ts_tablet_manager.cc:1367] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.005s
I20240430 17:02:51.875087 2390 raft_consensus.cc:348] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } } peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } }
I20240430 17:02:51.875808 2390 raft_consensus.cc:374] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:02:51.876104 2390 raft_consensus.cc:729] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be688569c19940fe856c07b5d80f7202, State: Initialized, Role: FOLLOWER
I20240430 17:02:51.876736 2390 consensus_queue.cc:260] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } } peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } }
I20240430 17:02:51.878813 2390 ts_tablet_manager.cc:1398] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240430 17:02:51.896404 2393 raft_consensus.cc:482] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:51.896782 2393 raft_consensus.cc:504] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } } peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } }
I20240430 17:02:51.898561 2393 leader_election.cc:288] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be688569c19940fe856c07b5d80f7202 (127.0.103.65:43615), 7ba0946bd92b47a6a06a1d09de274056 (127.0.103.66:46531)
I20240430 17:02:51.910449 2269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293b1c0771ea4a3cab5f279bdc069c72" candidate_uuid: "ba35a6d0a71d46909505e61bb9884415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ba0946bd92b47a6a06a1d09de274056" is_pre_election: true
I20240430 17:02:51.910609 2194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293b1c0771ea4a3cab5f279bdc069c72" candidate_uuid: "ba35a6d0a71d46909505e61bb9884415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be688569c19940fe856c07b5d80f7202" is_pre_election: true
I20240430 17:02:51.911384 2269 raft_consensus.cc:2454] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba35a6d0a71d46909505e61bb9884415 in term 0.
I20240430 17:02:51.911448 2194 raft_consensus.cc:2454] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba35a6d0a71d46909505e61bb9884415 in term 0.
I20240430 17:02:51.912761 2308 leader_election.cc:302] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ba0946bd92b47a6a06a1d09de274056, ba35a6d0a71d46909505e61bb9884415; no voters:
I20240430 17:02:51.913481 2393 raft_consensus.cc:2781] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:02:51.913830 2393 raft_consensus.cc:482] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:51.914135 2393 raft_consensus.cc:3037] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:51.918929 2393 raft_consensus.cc:504] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } } peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } }
I20240430 17:02:51.920284 2393 leader_election.cc:288] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [CANDIDATE]: Term 1 election: Requested vote from peers be688569c19940fe856c07b5d80f7202 (127.0.103.65:43615), 7ba0946bd92b47a6a06a1d09de274056 (127.0.103.66:46531)
I20240430 17:02:51.920950 2194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293b1c0771ea4a3cab5f279bdc069c72" candidate_uuid: "ba35a6d0a71d46909505e61bb9884415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be688569c19940fe856c07b5d80f7202"
I20240430 17:02:51.921177 2269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293b1c0771ea4a3cab5f279bdc069c72" candidate_uuid: "ba35a6d0a71d46909505e61bb9884415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ba0946bd92b47a6a06a1d09de274056"
I20240430 17:02:51.921403 2194 raft_consensus.cc:3037] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:51.921682 2269 raft_consensus.cc:3037] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:51.925856 2194 raft_consensus.cc:2454] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba35a6d0a71d46909505e61bb9884415 in term 1.
I20240430 17:02:51.926064 2269 raft_consensus.cc:2454] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba35a6d0a71d46909505e61bb9884415 in term 1.
I20240430 17:02:51.926842 2307 leader_election.cc:302] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba35a6d0a71d46909505e61bb9884415, be688569c19940fe856c07b5d80f7202; no voters:
I20240430 17:02:51.927557 2393 raft_consensus.cc:2781] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:51.928510 2393 raft_consensus.cc:686] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 1 LEADER]: Becoming Leader. State: Replica: ba35a6d0a71d46909505e61bb9884415, State: Running, Role: LEADER
I20240430 17:02:51.929090 2393 consensus_queue.cc:237] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } } peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } }
I20240430 17:02:51.936897 2091 catalog_manager.cc:5484] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba35a6d0a71d46909505e61bb9884415 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "ba35a6d0a71d46909505e61bb9884415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } health_report { overall_health: UNKNOWN } } }
I20240430 17:02:51.970475 2395 raft_consensus.cc:482] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:02:51.970952 2395 raft_consensus.cc:504] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } }
I20240430 17:02:51.972908 2395 leader_election.cc:288] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be688569c19940fe856c07b5d80f7202 (127.0.103.65:43615), ba35a6d0a71d46909505e61bb9884415 (127.0.103.67:41179)
I20240430 17:02:51.982780 2194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d26eca405014074995129f798290254" candidate_uuid: "7ba0946bd92b47a6a06a1d09de274056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be688569c19940fe856c07b5d80f7202" is_pre_election: true
I20240430 17:02:51.983119 2343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d26eca405014074995129f798290254" candidate_uuid: "7ba0946bd92b47a6a06a1d09de274056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba35a6d0a71d46909505e61bb9884415" is_pre_election: true
I20240430 17:02:51.983438 2194 raft_consensus.cc:2454] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ba0946bd92b47a6a06a1d09de274056 in term 0.
I20240430 17:02:51.983851 2343 raft_consensus.cc:2454] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ba0946bd92b47a6a06a1d09de274056 in term 0.
I20240430 17:02:51.984378 2233 leader_election.cc:302] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ba0946bd92b47a6a06a1d09de274056, be688569c19940fe856c07b5d80f7202; no voters:
I20240430 17:02:51.985008 2395 raft_consensus.cc:2781] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:02:51.985265 2395 raft_consensus.cc:482] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:02:51.985503 2395 raft_consensus.cc:3037] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:51.989962 2395 raft_consensus.cc:504] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } }
I20240430 17:02:51.991348 2395 leader_election.cc:288] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [CANDIDATE]: Term 1 election: Requested vote from peers be688569c19940fe856c07b5d80f7202 (127.0.103.65:43615), ba35a6d0a71d46909505e61bb9884415 (127.0.103.67:41179)
I20240430 17:02:51.992180 2194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d26eca405014074995129f798290254" candidate_uuid: "7ba0946bd92b47a6a06a1d09de274056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be688569c19940fe856c07b5d80f7202"
I20240430 17:02:51.992385 2343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d26eca405014074995129f798290254" candidate_uuid: "7ba0946bd92b47a6a06a1d09de274056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba35a6d0a71d46909505e61bb9884415"
I20240430 17:02:51.992621 2194 raft_consensus.cc:3037] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:51.992908 2343 raft_consensus.cc:3037] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:02:51.996902 2194 raft_consensus.cc:2454] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ba0946bd92b47a6a06a1d09de274056 in term 1.
I20240430 17:02:51.997265 2343 raft_consensus.cc:2454] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ba0946bd92b47a6a06a1d09de274056 in term 1.
I20240430 17:02:51.997778 2233 leader_election.cc:302] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ba0946bd92b47a6a06a1d09de274056, be688569c19940fe856c07b5d80f7202; no voters:
I20240430 17:02:51.998499 2395 raft_consensus.cc:2781] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:02:51.999598 2395 raft_consensus.cc:686] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 1 LEADER]: Becoming Leader. State: Replica: 7ba0946bd92b47a6a06a1d09de274056, State: Running, Role: LEADER
I20240430 17:02:52.000214 2395 consensus_queue.cc:237] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } }
I20240430 17:02:52.006837 2091 catalog_manager.cc:5484] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ba0946bd92b47a6a06a1d09de274056 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "7ba0946bd92b47a6a06a1d09de274056" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } health_report { overall_health: UNKNOWN } } }
I20240430 17:02:52.323532 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:02:52.325305 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:02:52.325613 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:02:52.421327 2393 consensus_queue.cc:1035] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:52.431463 2393 consensus_queue.cc:1035] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:52.479709 2395 consensus_queue.cc:1035] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:52.490082 2395 consensus_queue.cc:1035] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:02:54.326534 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:02:54.328337 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:02:54.328629 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:02:56.329517 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:02:56.331396 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:02:56.331673 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:02:58.332731 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:02:58.334915 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:02:58.335271 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:00.336150 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:00.337996 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:00.338325 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:02.339216 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:02.341077 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:02.341399 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:03.118739 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:03.126322 2422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:03.127153 2423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:03.127674 2425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:03.128561 413 server_base.cc:1034] running on GCE node
I20240430 17:03:03.130246 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:03:03.130458 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:03.130614 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496583130599 us; error 0 us; skew 500 ppm
I20240430 17:03:03.131086 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:03.140321 413 webserver.cc:460] Webserver started at http://127.0.103.68:46165/ using document root <none> and password file <none>
I20240430 17:03:03.140820 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:03.140995 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:03.141249 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:03.142627 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-3-root/instance:
uuid: "35e26bc7f5844831afc5310f2800a1db"
format_stamp: "Formatted at 2024-04-30 17:03:03 on dist-test-slave-x21r"
I20240430 17:03:03.147310 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240430 17:03:03.150645 2430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:03.151412 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240430 17:03:03.151710 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-3-root
uuid: "35e26bc7f5844831afc5310f2800a1db"
format_stamp: "Formatted at 2024-04-30 17:03:03 on dist-test-slave-x21r"
I20240430 17:03:03.151986 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:03.161203 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:03.162446 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:03.163976 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:03:03.166338 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:03:03.166551 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:03.166787 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:03:03.166949 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:03.207770 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:36915
I20240430 17:03:03.207871 2492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.68:36915 every 8 connection(s)
I20240430 17:03:03.223353 2493 heartbeater.cc:347] Connected to a master server at 127.0.103.126:39231
I20240430 17:03:03.223732 2493 heartbeater.cc:464] Registering TS with master...
I20240430 17:03:03.224465 2493 heartbeater.cc:511] Master 127.0.103.126:39231 requested a full tablet report, sending...
I20240430 17:03:03.226320 2091 ts_manager.cc:194] Registered new tserver with Master: 35e26bc7f5844831afc5310f2800a1db (127.0.103.68:36915)
I20240430 17:03:03.227823 2091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49470
I20240430 17:03:03.381736 2269 consensus_queue.cc:237] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: true } }
I20240430 17:03:03.390638 2194 raft_consensus.cc:1261] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 1 FOLLOWER]: Refusing update from remote peer 7ba0946bd92b47a6a06a1d09de274056: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:03:03.390545 2343 raft_consensus.cc:1261] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [term 1 FOLLOWER]: Refusing update from remote peer 7ba0946bd92b47a6a06a1d09de274056: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:03:03.392125 2419 consensus_queue.cc:1035] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 17:03:03.393110 2500 consensus_queue.cc:1035] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240430 17:03:03.400521 2233 consensus_peers.cc:485] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 -> Peer 35e26bc7f5844831afc5310f2800a1db (127.0.103.68:36915): Couldn't send request to peer 35e26bc7f5844831afc5310f2800a1db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d26eca405014074995129f798290254. This is attempt 1: this message will repeat every 5th retry.
I20240430 17:03:03.401965 2500 raft_consensus.cc:2932] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 35e26bc7f5844831afc5310f2800a1db (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: true } } }
I20240430 17:03:03.403620 2343 raft_consensus.cc:2932] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 35e26bc7f5844831afc5310f2800a1db (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: true } } }
I20240430 17:03:03.405417 2194 raft_consensus.cc:2932] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 35e26bc7f5844831afc5310f2800a1db (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: true } } }
I20240430 17:03:03.415678 2091 catalog_manager.cc:5484] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 reported cstate change: config changed from index -1 to 2, NON_VOTER 35e26bc7f5844831afc5310f2800a1db (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "7ba0946bd92b47a6a06a1d09de274056" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: true } } }
I20240430 17:03:03.860836 2511 ts_tablet_manager.cc:902] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db: Initiating tablet copy from peer 7ba0946bd92b47a6a06a1d09de274056 (127.0.103.66:46531)
I20240430 17:03:03.864682 2511 tablet_copy_client.cc:308] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:46531
I20240430 17:03:03.874303 2279 tablet_copy_service.cc:140] P 7ba0946bd92b47a6a06a1d09de274056: Received BeginTabletCopySession request for tablet 0d26eca405014074995129f798290254 from peer 35e26bc7f5844831afc5310f2800a1db ({username='slave'} at 127.0.0.1:40916)
I20240430 17:03:03.874769 2279 tablet_copy_service.cc:161] P 7ba0946bd92b47a6a06a1d09de274056: Beginning new tablet copy session on tablet 0d26eca405014074995129f798290254 from peer 35e26bc7f5844831afc5310f2800a1db at {username='slave'} at 127.0.0.1:40916: session id = 35e26bc7f5844831afc5310f2800a1db-0d26eca405014074995129f798290254
I20240430 17:03:03.881127 2279 tablet_copy_source_session.cc:205] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056: Tablet Copy: opened 0 blocks and 1 log segments
I20240430 17:03:03.883626 2511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d26eca405014074995129f798290254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:03:03.894765 2511 tablet_copy_client.cc:783] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db: tablet copy: Starting download of 0 data blocks...
I20240430 17:03:03.895311 2511 tablet_copy_client.cc:647] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db: tablet copy: Starting download of 1 WAL segments...
I20240430 17:03:03.898519 2511 tablet_copy_client.cc:523] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240430 17:03:03.904908 2511 tablet_bootstrap.cc:492] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db: Bootstrap starting.
I20240430 17:03:03.919529 2511 tablet_bootstrap.cc:492] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240430 17:03:03.920282 2511 tablet_bootstrap.cc:492] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db: Bootstrap complete.
I20240430 17:03:03.921087 2511 ts_tablet_manager.cc:1367] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20240430 17:03:03.923540 2511 raft_consensus.cc:348] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: true } }
I20240430 17:03:03.924252 2511 raft_consensus.cc:729] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 35e26bc7f5844831afc5310f2800a1db, State: Initialized, Role: LEARNER
I20240430 17:03:03.924901 2511 consensus_queue.cc:260] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: true } }
I20240430 17:03:03.926733 2493 heartbeater.cc:503] Master 127.0.103.126:39231 was elected leader, sending a full tablet report...
I20240430 17:03:03.927275 2511 ts_tablet_manager.cc:1398] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20240430 17:03:03.928985 2279 tablet_copy_service.cc:342] P 7ba0946bd92b47a6a06a1d09de274056: Request end of tablet copy session 35e26bc7f5844831afc5310f2800a1db-0d26eca405014074995129f798290254 received from {username='slave'} at 127.0.0.1:40916
I20240430 17:03:03.929363 2279 tablet_copy_service.cc:433] P 7ba0946bd92b47a6a06a1d09de274056: ending tablet copy session 35e26bc7f5844831afc5310f2800a1db-0d26eca405014074995129f798290254 on tablet 0d26eca405014074995129f798290254 with peer 35e26bc7f5844831afc5310f2800a1db
I20240430 17:03:04.342211 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:04.344343 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:04.344691 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:04.441310 2468 raft_consensus.cc:1203] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240430 17:03:04.936795 2520 raft_consensus.cc:1050] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056: attempting to promote NON_VOTER 35e26bc7f5844831afc5310f2800a1db to VOTER
I20240430 17:03:04.938673 2520 consensus_queue.cc:237] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false } }
I20240430 17:03:04.945034 2468 raft_consensus.cc:1261] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db [term 1 LEARNER]: Refusing update from remote peer 7ba0946bd92b47a6a06a1d09de274056: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240430 17:03:04.945149 2343 raft_consensus.cc:1261] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [term 1 FOLLOWER]: Refusing update from remote peer 7ba0946bd92b47a6a06a1d09de274056: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240430 17:03:04.946846 2194 raft_consensus.cc:1261] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 1 FOLLOWER]: Refusing update from remote peer 7ba0946bd92b47a6a06a1d09de274056: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240430 17:03:04.946617 2523 consensus_queue.cc:1035] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 17:03:04.947890 2520 consensus_queue.cc:1035] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 17:03:04.948827 2519 consensus_queue.cc:1035] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 17:03:04.959880 2519 raft_consensus.cc:2932] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 35e26bc7f5844831afc5310f2800a1db (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false } } }
I20240430 17:03:04.961434 2468 raft_consensus.cc:2932] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 35e26bc7f5844831afc5310f2800a1db (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false } } }
I20240430 17:03:04.963167 2343 raft_consensus.cc:2932] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 35e26bc7f5844831afc5310f2800a1db (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false } } }
I20240430 17:03:04.969528 2194 raft_consensus.cc:2932] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 35e26bc7f5844831afc5310f2800a1db (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false } } }
I20240430 17:03:04.972726 2090 catalog_manager.cc:5484] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 reported cstate change: config changed from index 2 to 3, 35e26bc7f5844831afc5310f2800a1db (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ba0946bd92b47a6a06a1d09de274056" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba35a6d0a71d46909505e61bb9884415" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240430 17:03:04.994508 2268 consensus_queue.cc:237] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false } }
I20240430 17:03:04.998778 2468 raft_consensus.cc:1261] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db [term 1 FOLLOWER]: Refusing update from remote peer 7ba0946bd92b47a6a06a1d09de274056: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240430 17:03:04.999892 2523 consensus_queue.cc:1035] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240430 17:03:05.000276 2194 raft_consensus.cc:1261] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 1 FOLLOWER]: Refusing update from remote peer 7ba0946bd92b47a6a06a1d09de274056: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240430 17:03:05.001391 2519 consensus_queue.cc:1035] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240430 17:03:05.006938 2520 raft_consensus.cc:2932] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ba35a6d0a71d46909505e61bb9884415 (127.0.103.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false } } }
I20240430 17:03:05.008772 2468 raft_consensus.cc:2932] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ba35a6d0a71d46909505e61bb9884415 (127.0.103.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false } } }
I20240430 17:03:05.008670 2194 raft_consensus.cc:2932] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ba35a6d0a71d46909505e61bb9884415 (127.0.103.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false } } }
I20240430 17:03:05.018090 2078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d26eca405014074995129f798290254 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240430 17:03:05.021730 2089 catalog_manager.cc:5484] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db reported cstate change: config changed from index 3 to 4, VOTER ba35a6d0a71d46909505e61bb9884415 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "7ba0946bd92b47a6a06a1d09de274056" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be688569c19940fe856c07b5d80f7202" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43615 } } peers { permanent_uuid: "7ba0946bd92b47a6a06a1d09de274056" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46531 } } peers { permanent_uuid: "35e26bc7f5844831afc5310f2800a1db" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36915 } attrs { promote: false } } }
I20240430 17:03:05.031275 2333 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d26eca405014074995129f798290254 with delete_type TABLET_DATA_TOMBSTONED (TS ba35a6d0a71d46909505e61bb9884415 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59152
I20240430 17:03:05.033494 2533 tablet_replica.cc:330] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415: stopping tablet replica
I20240430 17:03:05.034252 2533 raft_consensus.cc:2229] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:03:05.035019 2533 raft_consensus.cc:2258] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:03:05.037806 2533 ts_tablet_manager.cc:1875] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240430 17:03:05.047670 2533 ts_tablet_manager.cc:1888] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240430 17:03:05.048000 2533 log.cc:1196] T 0d26eca405014074995129f798290254 P ba35a6d0a71d46909505e61bb9884415: Deleting WAL directory at /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714496547126677-413-0/minicluster-data/ts-2-root/wals/0d26eca405014074995129f798290254
I20240430 17:03:05.049348 2077 catalog_manager.cc:4830] TS ba35a6d0a71d46909505e61bb9884415 (127.0.103.67:41179): tablet 0d26eca405014074995129f798290254 (table test-workload [id=4a4b2d99900349fd8c6834d344a2cc3e]) successfully deleted
I20240430 17:03:06.345616 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:06.347602 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:06.347883 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:08.348877 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:08.350912 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:08.351256 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:10.352136 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:10.353987 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:10.354249 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:12.355032 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:12.356791 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:12.357043 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:14.357787 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:14.359556 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:14.359833 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:16.360536 2145 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:16.362309 2145 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:16.362572 2145 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:17.050784 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20240430 17:03:17.078536 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:03:17.079202 413 tablet_replica.cc:330] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202: stopping tablet replica
I20240430 17:03:17.079780 413 raft_consensus.cc:2229] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:03:17.080304 413 raft_consensus.cc:2258] T 293b1c0771ea4a3cab5f279bdc069c72 P be688569c19940fe856c07b5d80f7202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:03:17.082278 413 tablet_replica.cc:330] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202: stopping tablet replica
I20240430 17:03:17.082702 413 raft_consensus.cc:2229] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:03:17.083523 413 raft_consensus.cc:2258] T 0d26eca405014074995129f798290254 P be688569c19940fe856c07b5d80f7202 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 17:03:17.085911 2233 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 9 similar messages]
W20240430 17:03:17.088943 2233 consensus_peers.cc:485] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 -> Peer be688569c19940fe856c07b5d80f7202 (127.0.103.65:43615): Couldn't send request to peer be688569c19940fe856c07b5d80f7202. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240430 17:03:17.105664 413 tablet_server.cc:194] TabletServer@127.0.103.65:0 shutdown complete.
I20240430 17:03:17.117609 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20240430 17:03:17.138602 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:03:17.139212 413 tablet_replica.cc:330] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056: stopping tablet replica
I20240430 17:03:17.139737 413 raft_consensus.cc:2229] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:03:17.140136 413 raft_consensus.cc:2258] T 293b1c0771ea4a3cab5f279bdc069c72 P 7ba0946bd92b47a6a06a1d09de274056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:03:17.142051 413 tablet_replica.cc:330] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056: stopping tablet replica
I20240430 17:03:17.142452 413 raft_consensus.cc:2229] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:03:17.143265 413 raft_consensus.cc:2258] T 0d26eca405014074995129f798290254 P 7ba0946bd92b47a6a06a1d09de274056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:03:17.164501 413 tablet_server.cc:194] TabletServer@127.0.103.66:0 shutdown complete.
I20240430 17:03:17.175701 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20240430 17:03:17.200546 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:03:17.201164 413 tablet_replica.cc:330] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415: stopping tablet replica
I20240430 17:03:17.201625 413 raft_consensus.cc:2229] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:03:17.202401 413 raft_consensus.cc:2258] T 293b1c0771ea4a3cab5f279bdc069c72 P ba35a6d0a71d46909505e61bb9884415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:03:17.222803 413 tablet_server.cc:194] TabletServer@127.0.103.67:0 shutdown complete.
I20240430 17:03:17.233610 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20240430 17:03:17.254419 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:03:17.255053 413 tablet_replica.cc:330] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db: stopping tablet replica
I20240430 17:03:17.255628 413 raft_consensus.cc:2229] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:03:17.256274 413 raft_consensus.cc:2258] T 0d26eca405014074995129f798290254 P 35e26bc7f5844831afc5310f2800a1db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:03:17.274605 413 tablet_server.cc:194] TabletServer@127.0.103.68:0 shutdown complete.
I20240430 17:03:17.283823 413 master.cc:537] Master@127.0.103.126:39231 shutting down...
I20240430 17:03:17.300391 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:03:17.300957 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:03:17.301261 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P f5afb6fc1dc24fcd9bdbe41d8e27c965: stopping tablet replica
I20240430 17:03:17.318660 413 master.cc:552] Master@127.0.103.126:39231 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (27243 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240430 17:03:17.353035 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:42577
I20240430 17:03:17.354043 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:17.360139 2541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:17.360992 2542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:17.361761 2544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:17.362375 413 server_base.cc:1034] running on GCE node
I20240430 17:03:17.363322 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:03:17.363518 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:17.363657 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496597363646 us; error 0 us; skew 500 ppm
I20240430 17:03:17.364054 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:17.366266 413 webserver.cc:460] Webserver started at http://127.0.103.126:39955/ using document root <none> and password file <none>
I20240430 17:03:17.366685 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:17.366837 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:17.367045 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:17.368253 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714496547126677-413-0/minicluster-data/master-0-root/instance:
uuid: "50291b4c0ba940e89db15c5c24857ca5"
format_stamp: "Formatted at 2024-04-30 17:03:17 on dist-test-slave-x21r"
I20240430 17:03:17.372402 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240430 17:03:17.375491 2549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:17.376152 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240430 17:03:17.376376 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714496547126677-413-0/minicluster-data/master-0-root
uuid: "50291b4c0ba940e89db15c5c24857ca5"
format_stamp: "Formatted at 2024-04-30 17:03:17 on dist-test-slave-x21r"
I20240430 17:03:17.376614 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714496547126677-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714496547126677-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714496547126677-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:17.411789 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:17.412909 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:17.448094 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:42577
I20240430 17:03:17.448189 2600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.126:42577 every 8 connection(s)
I20240430 17:03:17.451846 2601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:03:17.462086 2601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5: Bootstrap starting.
I20240430 17:03:17.466054 2601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5: Neither blocks nor log segments found. Creating new log.
I20240430 17:03:17.469836 2601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5: No bootstrap required, opened a new log
I20240430 17:03:17.471680 2601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50291b4c0ba940e89db15c5c24857ca5" member_type: VOTER }
I20240430 17:03:17.472137 2601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:03:17.472366 2601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50291b4c0ba940e89db15c5c24857ca5, State: Initialized, Role: FOLLOWER
I20240430 17:03:17.472925 2601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50291b4c0ba940e89db15c5c24857ca5" member_type: VOTER }
I20240430 17:03:17.473418 2601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:03:17.473630 2601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:03:17.473891 2601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:03:17.477918 2601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50291b4c0ba940e89db15c5c24857ca5" member_type: VOTER }
I20240430 17:03:17.478418 2601 leader_election.cc:302] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50291b4c0ba940e89db15c5c24857ca5; no voters:
I20240430 17:03:17.479539 2601 leader_election.cc:288] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:03:17.480084 2604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:03:17.481462 2604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 50291b4c0ba940e89db15c5c24857ca5, State: Running, Role: LEADER
I20240430 17:03:17.490999 2604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50291b4c0ba940e89db15c5c24857ca5" member_type: VOTER }
I20240430 17:03:17.492226 2601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 17:03:17.500563 2606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50291b4c0ba940e89db15c5c24857ca5. Latest consensus state: current_term: 1 leader_uuid: "50291b4c0ba940e89db15c5c24857ca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50291b4c0ba940e89db15c5c24857ca5" member_type: VOTER } }
I20240430 17:03:17.501271 2606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [sys.catalog]: This master's current role is: LEADER
I20240430 17:03:17.502764 2607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50291b4c0ba940e89db15c5c24857ca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50291b4c0ba940e89db15c5c24857ca5" member_type: VOTER } }
I20240430 17:03:17.503084 2614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 17:03:17.503255 2607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [sys.catalog]: This master's current role is: LEADER
I20240430 17:03:17.510949 2614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 17:03:17.512225 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240430 17:03:17.520182 2614 catalog_manager.cc:1347] Generated new cluster ID: 258911055e3643e7a7ffc0127f3993f1
I20240430 17:03:17.520411 2614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 17:03:17.532958 2614 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 17:03:17.534145 2614 catalog_manager.cc:1504] Loading token signing keys...
I20240430 17:03:17.545065 2614 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5: Generated new TSK 0
I20240430 17:03:17.545567 2614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 17:03:17.577366 413 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.000199982s
I20240430 17:03:29.670256 413 master.cc:537] Master@127.0.103.126:42577 shutting down...
I20240430 17:03:29.694649 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:03:29.695475 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:03:29.695823 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P 50291b4c0ba940e89db15c5c24857ca5: stopping tablet replica
I20240430 17:03:29.715760 413 master.cc:552] Master@127.0.103.126:42577 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12394 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240430 17:03:29.748427 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:46247
I20240430 17:03:29.749498 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:29.755666 2625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:29.756420 2626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:29.757118 2628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:29.757799 413 server_base.cc:1034] running on GCE node
I20240430 17:03:29.758466 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:03:29.758651 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:29.758791 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496609758776 us; error 0 us; skew 500 ppm
I20240430 17:03:29.759272 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:29.765183 413 webserver.cc:460] Webserver started at http://127.0.103.126:39285/ using document root <none> and password file <none>
I20240430 17:03:29.765648 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:29.765816 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:29.766067 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:29.767331 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/master-0-root/instance:
uuid: "bb6f68ae7c3a400cbd26352475877f0d"
format_stamp: "Formatted at 2024-04-30 17:03:29 on dist-test-slave-x21r"
I20240430 17:03:29.771795 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240430 17:03:29.775019 2633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:29.775756 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240430 17:03:29.776000 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/master-0-root
uuid: "bb6f68ae7c3a400cbd26352475877f0d"
format_stamp: "Formatted at 2024-04-30 17:03:29 on dist-test-slave-x21r"
I20240430 17:03:29.776254 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:29.799700 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:29.800856 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:29.836272 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:46247
I20240430 17:03:29.836369 2684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.126:46247 every 8 connection(s)
I20240430 17:03:29.840096 2685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:03:29.850637 2685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d: Bootstrap starting.
I20240430 17:03:29.854740 2685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d: Neither blocks nor log segments found. Creating new log.
I20240430 17:03:29.858628 2685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d: No bootstrap required, opened a new log
I20240430 17:03:29.860610 2685 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6f68ae7c3a400cbd26352475877f0d" member_type: VOTER }
I20240430 17:03:29.861073 2685 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:03:29.861287 2685 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6f68ae7c3a400cbd26352475877f0d, State: Initialized, Role: FOLLOWER
I20240430 17:03:29.861786 2685 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6f68ae7c3a400cbd26352475877f0d" member_type: VOTER }
I20240430 17:03:29.862236 2685 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:03:29.862432 2685 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:03:29.862677 2685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:03:29.866899 2685 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6f68ae7c3a400cbd26352475877f0d" member_type: VOTER }
I20240430 17:03:29.867403 2685 leader_election.cc:302] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb6f68ae7c3a400cbd26352475877f0d; no voters:
I20240430 17:03:29.868425 2685 leader_election.cc:288] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:03:29.868817 2688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:03:29.870019 2688 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [term 1 LEADER]: Becoming Leader. State: Replica: bb6f68ae7c3a400cbd26352475877f0d, State: Running, Role: LEADER
I20240430 17:03:29.870671 2688 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6f68ae7c3a400cbd26352475877f0d" member_type: VOTER }
I20240430 17:03:29.871243 2685 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [sys.catalog]: configured and running, proceeding with master startup.
I20240430 17:03:29.873484 2689 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb6f68ae7c3a400cbd26352475877f0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6f68ae7c3a400cbd26352475877f0d" member_type: VOTER } }
I20240430 17:03:29.874198 2689 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [sys.catalog]: This master's current role is: LEADER
I20240430 17:03:29.873492 2690 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb6f68ae7c3a400cbd26352475877f0d. Latest consensus state: current_term: 1 leader_uuid: "bb6f68ae7c3a400cbd26352475877f0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6f68ae7c3a400cbd26352475877f0d" member_type: VOTER } }
I20240430 17:03:29.875341 2690 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [sys.catalog]: This master's current role is: LEADER
I20240430 17:03:29.877549 2693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 17:03:29.882874 2693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 17:03:29.888022 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240430 17:03:29.892354 2693 catalog_manager.cc:1347] Generated new cluster ID: 2d3040019e3d48d58c08736b35a8c892
I20240430 17:03:29.892606 2693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 17:03:29.906045 2693 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 17:03:29.907210 2693 catalog_manager.cc:1504] Loading token signing keys...
I20240430 17:03:29.920859 2693 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d: Generated new TSK 0
I20240430 17:03:29.921348 2693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 17:03:29.954284 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:29.960994 2706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:29.961921 2707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:29.962725 2709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:29.963552 413 server_base.cc:1034] running on GCE node
I20240430 17:03:29.964265 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:03:29.964476 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:29.964620 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496609964605 us; error 0 us; skew 500 ppm
I20240430 17:03:29.965118 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:29.967316 413 webserver.cc:460] Webserver started at http://127.0.103.65:34999/ using document root <none> and password file <none>
I20240430 17:03:29.967748 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:29.967926 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:29.968160 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:29.969273 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-0-root/instance:
uuid: "e82205ad00da438a8b5f79c99f721a87"
format_stamp: "Formatted at 2024-04-30 17:03:29 on dist-test-slave-x21r"
I20240430 17:03:29.973557 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240430 17:03:29.976589 2714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:29.977356 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240430 17:03:29.977602 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-0-root
uuid: "e82205ad00da438a8b5f79c99f721a87"
format_stamp: "Formatted at 2024-04-30 17:03:29 on dist-test-slave-x21r"
I20240430 17:03:29.977854 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:29.998970 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:30.000078 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:30.001442 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:03:30.003701 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:03:30.003882 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:30.004122 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:03:30.004264 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:30.042181 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:36347
I20240430 17:03:30.042265 2776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.65:36347 every 8 connection(s)
I20240430 17:03:30.047448 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:30.057250 2782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:30.058308 2781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:30.058909 2784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:30.059777 413 server_base.cc:1034] running on GCE node
I20240430 17:03:30.060714 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:03:30.060981 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:30.061159 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496610061142 us; error 0 us; skew 500 ppm
I20240430 17:03:30.061892 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:30.064152 2777 heartbeater.cc:347] Connected to a master server at 127.0.103.126:46247
I20240430 17:03:30.064474 2777 heartbeater.cc:464] Registering TS with master...
I20240430 17:03:30.064827 413 webserver.cc:460] Webserver started at http://127.0.103.66:35989/ using document root <none> and password file <none>
I20240430 17:03:30.065212 2777 heartbeater.cc:511] Master 127.0.103.126:46247 requested a full tablet report, sending...
I20240430 17:03:30.065289 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:30.065594 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:30.065842 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:30.067138 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-1-root/instance:
uuid: "b34acdbb5c9147a3a151bac21fcaa1ac"
format_stamp: "Formatted at 2024-04-30 17:03:30 on dist-test-slave-x21r"
I20240430 17:03:30.067188 2650 ts_manager.cc:194] Registered new tserver with Master: e82205ad00da438a8b5f79c99f721a87 (127.0.103.65:36347)
I20240430 17:03:30.068965 2650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60664
I20240430 17:03:30.071736 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240430 17:03:30.074892 2789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:30.075589 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240430 17:03:30.075838 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-1-root
uuid: "b34acdbb5c9147a3a151bac21fcaa1ac"
format_stamp: "Formatted at 2024-04-30 17:03:30 on dist-test-slave-x21r"
I20240430 17:03:30.076103 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:30.097419 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:30.098560 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:30.099923 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:03:30.102037 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:03:30.102226 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:30.102442 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:03:30.102588 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:30.139987 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:43279
I20240430 17:03:30.140084 2851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.66:43279 every 8 connection(s)
I20240430 17:03:30.144451 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:30.153853 2855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:30.155530 2856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:30.156129 2852 heartbeater.cc:347] Connected to a master server at 127.0.103.126:46247
I20240430 17:03:30.156683 2852 heartbeater.cc:464] Registering TS with master...
I20240430 17:03:30.157781 2852 heartbeater.cc:511] Master 127.0.103.126:46247 requested a full tablet report, sending...
I20240430 17:03:30.160583 2650 ts_manager.cc:194] Registered new tserver with Master: b34acdbb5c9147a3a151bac21fcaa1ac (127.0.103.66:43279)
I20240430 17:03:30.161936 2650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60670
W20240430 17:03:30.162125 2858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:30.163065 413 server_base.cc:1034] running on GCE node
I20240430 17:03:30.164078 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:03:30.164327 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:30.164462 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496610164451 us; error 0 us; skew 500 ppm
I20240430 17:03:30.164908 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:30.167088 413 webserver.cc:460] Webserver started at http://127.0.103.67:45007/ using document root <none> and password file <none>
I20240430 17:03:30.167523 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:30.167677 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:30.167881 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:30.168932 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-2-root/instance:
uuid: "1bb6aedc94ce4129b0e2a69c3997d116"
format_stamp: "Formatted at 2024-04-30 17:03:30 on dist-test-slave-x21r"
I20240430 17:03:30.173038 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20240430 17:03:30.175925 2863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:30.176725 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240430 17:03:30.176973 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-2-root
uuid: "1bb6aedc94ce4129b0e2a69c3997d116"
format_stamp: "Formatted at 2024-04-30 17:03:30 on dist-test-slave-x21r"
I20240430 17:03:30.177201 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714496547126677-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:30.190902 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:30.191988 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:30.193194 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:03:30.195498 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:03:30.195662 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:30.195835 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:03:30.195961 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:30.232589 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:45347
I20240430 17:03:30.232676 2925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.67:45347 every 8 connection(s)
I20240430 17:03:30.244882 2926 heartbeater.cc:347] Connected to a master server at 127.0.103.126:46247
I20240430 17:03:30.245201 2926 heartbeater.cc:464] Registering TS with master...
I20240430 17:03:30.245896 2926 heartbeater.cc:511] Master 127.0.103.126:46247 requested a full tablet report, sending...
I20240430 17:03:30.247673 2650 ts_manager.cc:194] Registered new tserver with Master: 1bb6aedc94ce4129b0e2a69c3997d116 (127.0.103.67:45347)
I20240430 17:03:30.247804 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012212238s
I20240430 17:03:30.248953 2650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60682
I20240430 17:03:31.071409 2777 heartbeater.cc:503] Master 127.0.103.126:46247 was elected leader, sending a full tablet report...
I20240430 17:03:31.164517 2852 heartbeater.cc:503] Master 127.0.103.126:46247 was elected leader, sending a full tablet report...
I20240430 17:03:31.251662 2926 heartbeater.cc:503] Master 127.0.103.126:46247 was elected leader, sending a full tablet report...
I20240430 17:03:31.887831 2703 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240430 17:03:32.117883 2922 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1f20 after lost signal to thread 416
W20240430 17:03:32.118723 2922 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1600 after lost signal to thread 2684
W20240430 17:03:32.119700 2922 debug-util.cc:398] Leaking SignalData structure 0x7b08001cc180 after lost signal to thread 2776
W20240430 17:03:32.120579 2922 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 2851
W20240430 17:03:32.121358 2922 debug-util.cc:398] Leaking SignalData structure 0x7b0800210840 after lost signal to thread 2925
I20240430 17:03:33.888940 2703 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:35.890076 2703 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:37.891371 2703 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:39.892485 2703 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:41.893452 2703 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:42.547783 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20240430 17:03:42.576053 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:03:42.592896 413 tablet_server.cc:194] TabletServer@127.0.103.65:0 shutdown complete.
I20240430 17:03:42.602290 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20240430 17:03:42.623335 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:03:42.638891 413 tablet_server.cc:194] TabletServer@127.0.103.66:0 shutdown complete.
I20240430 17:03:42.647588 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20240430 17:03:42.665537 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:03:42.697093 413 tablet_server.cc:194] TabletServer@127.0.103.67:0 shutdown complete.
I20240430 17:03:42.705443 413 master.cc:537] Master@127.0.103.126:46247 shutting down...
I20240430 17:03:42.725306 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:03:42.726027 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:03:42.726341 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P bb6f68ae7c3a400cbd26352475877f0d: stopping tablet replica
I20240430 17:03:42.743898 413 master.cc:552] Master@127.0.103.126:46247 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (13024 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240430 17:03:42.771973 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:34603
I20240430 17:03:42.773006 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:42.778784 2931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:42.779608 2932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:42.780611 2934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:42.781348 413 server_base.cc:1034] running on GCE node
I20240430 17:03:42.782115 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:03:42.782307 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:42.782459 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496622782440 us; error 0 us; skew 500 ppm
I20240430 17:03:42.782893 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:42.785305 413 webserver.cc:460] Webserver started at http://127.0.103.126:38721/ using document root <none> and password file <none>
I20240430 17:03:42.785759 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:42.785933 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:42.786185 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:42.787396 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/master-0-root/instance:
uuid: "92c52a11991d4a21bb9059571adc5dae"
format_stamp: "Formatted at 2024-04-30 17:03:42 on dist-test-slave-x21r"
I20240430 17:03:42.791600 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240430 17:03:42.794591 2939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:42.795298 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240430 17:03:42.795544 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/master-0-root
uuid: "92c52a11991d4a21bb9059571adc5dae"
format_stamp: "Formatted at 2024-04-30 17:03:42 on dist-test-slave-x21r"
I20240430 17:03:42.795799 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:42.807984 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:42.809065 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:42.844487 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:34603
I20240430 17:03:42.844578 2990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.126:34603 every 8 connection(s)
I20240430 17:03:42.848295 2991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:03:42.858971 2991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae: Bootstrap starting.
I20240430 17:03:42.862993 2991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae: Neither blocks nor log segments found. Creating new log.
I20240430 17:03:42.866842 2991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae: No bootstrap required, opened a new log
I20240430 17:03:42.868724 2991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c52a11991d4a21bb9059571adc5dae" member_type: VOTER }
I20240430 17:03:42.869201 2991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:03:42.869382 2991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c52a11991d4a21bb9059571adc5dae, State: Initialized, Role: FOLLOWER
I20240430 17:03:42.869907 2991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c52a11991d4a21bb9059571adc5dae" member_type: VOTER }
I20240430 17:03:42.870321 2991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:03:42.870508 2991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:03:42.870721 2991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:03:42.874794 2991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c52a11991d4a21bb9059571adc5dae" member_type: VOTER }
I20240430 17:03:42.875290 2991 leader_election.cc:302] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92c52a11991d4a21bb9059571adc5dae; no voters:
I20240430 17:03:42.876332 2991 leader_election.cc:288] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:03:42.876689 2994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:03:42.877925 2994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [term 1 LEADER]: Becoming Leader. State: Replica: 92c52a11991d4a21bb9059571adc5dae, State: Running, Role: LEADER
I20240430 17:03:42.878491 2994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c52a11991d4a21bb9059571adc5dae" member_type: VOTER }
I20240430 17:03:42.879137 2991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [sys.catalog]: configured and running, proceeding with master startup.
I20240430 17:03:42.884191 2996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92c52a11991d4a21bb9059571adc5dae. Latest consensus state: current_term: 1 leader_uuid: "92c52a11991d4a21bb9059571adc5dae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c52a11991d4a21bb9059571adc5dae" member_type: VOTER } }
I20240430 17:03:42.884538 2995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92c52a11991d4a21bb9059571adc5dae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c52a11991d4a21bb9059571adc5dae" member_type: VOTER } }
I20240430 17:03:42.884953 2996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [sys.catalog]: This master's current role is: LEADER
I20240430 17:03:42.885049 2995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [sys.catalog]: This master's current role is: LEADER
I20240430 17:03:42.886847 3003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 17:03:42.891417 3003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 17:03:42.893334 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240430 17:03:42.899622 3003 catalog_manager.cc:1347] Generated new cluster ID: e27f715cfe914369a0237ccf50d0dd8b
I20240430 17:03:42.899888 3003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 17:03:42.944613 3003 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 17:03:42.945814 3003 catalog_manager.cc:1504] Loading token signing keys...
I20240430 17:03:42.957155 3003 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae: Generated new TSK 0
I20240430 17:03:42.957715 3003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 17:03:43.023679 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:43.030102 3012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:43.031283 3013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:43.032001 3015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:43.032806 413 server_base.cc:1034] running on GCE node
I20240430 17:03:43.033551 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:03:43.033723 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:43.033840 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496623033830 us; error 0 us; skew 500 ppm
I20240430 17:03:43.034309 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:43.036617 413 webserver.cc:460] Webserver started at http://127.0.103.65:38267/ using document root <none> and password file <none>
I20240430 17:03:43.037021 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:43.037170 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:43.037376 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:43.038488 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-0-root/instance:
uuid: "4013dd8c69644fe0b0586bd800b734ac"
format_stamp: "Formatted at 2024-04-30 17:03:43 on dist-test-slave-x21r"
I20240430 17:03:43.042611 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240430 17:03:43.045640 3020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:43.046423 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240430 17:03:43.046680 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-0-root
uuid: "4013dd8c69644fe0b0586bd800b734ac"
format_stamp: "Formatted at 2024-04-30 17:03:43 on dist-test-slave-x21r"
I20240430 17:03:43.046940 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:43.066558 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:43.067632 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:43.069005 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:03:43.071300 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:03:43.071489 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:43.071717 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:03:43.071870 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:43.110409 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:45679
I20240430 17:03:43.110493 3082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.65:45679 every 8 connection(s)
I20240430 17:03:43.114967 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:43.123018 3087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:43.124167 3088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:43.127054 3090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:43.127782 3083 heartbeater.cc:347] Connected to a master server at 127.0.103.126:34603
I20240430 17:03:43.128087 413 server_base.cc:1034] running on GCE node
I20240430 17:03:43.128186 3083 heartbeater.cc:464] Registering TS with master...
I20240430 17:03:43.129113 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240430 17:03:43.129185 3083 heartbeater.cc:511] Master 127.0.103.126:34603 requested a full tablet report, sending...
W20240430 17:03:43.129457 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:43.129690 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496623129671 us; error 0 us; skew 500 ppm
I20240430 17:03:43.130311 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:43.131561 2956 ts_manager.cc:194] Registered new tserver with Master: 4013dd8c69644fe0b0586bd800b734ac (127.0.103.65:45679)
I20240430 17:03:43.133098 413 webserver.cc:460] Webserver started at http://127.0.103.66:43381/ using document root <none> and password file <none>
I20240430 17:03:43.133718 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:43.133935 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:43.134030 2956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46692
I20240430 17:03:43.134272 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:43.135864 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-1-root/instance:
uuid: "a5d45b3b171246f49df426276bc7c2c4"
format_stamp: "Formatted at 2024-04-30 17:03:43 on dist-test-slave-x21r"
I20240430 17:03:43.140172 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240430 17:03:43.143180 3095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:43.143852 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240430 17:03:43.144109 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-1-root
uuid: "a5d45b3b171246f49df426276bc7c2c4"
format_stamp: "Formatted at 2024-04-30 17:03:43 on dist-test-slave-x21r"
I20240430 17:03:43.144341 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:43.168962 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:43.169981 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:43.171272 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:03:43.173444 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:03:43.173621 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:43.173836 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:03:43.174011 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:43.212131 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:35499
I20240430 17:03:43.212217 3157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.66:35499 every 8 connection(s)
I20240430 17:03:43.216473 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:43.223340 3161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:43.224359 3162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:43.225647 3164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:43.227084 413 server_base.cc:1034] running on GCE node
I20240430 17:03:43.227958 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240430 17:03:43.228029 3158 heartbeater.cc:347] Connected to a master server at 127.0.103.126:34603
W20240430 17:03:43.228179 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:43.228456 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496623228441 us; error 0 us; skew 500 ppm
I20240430 17:03:43.228495 3158 heartbeater.cc:464] Registering TS with master...
I20240430 17:03:43.229038 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:43.229285 3158 heartbeater.cc:511] Master 127.0.103.126:34603 requested a full tablet report, sending...
I20240430 17:03:43.231163 2956 ts_manager.cc:194] Registered new tserver with Master: a5d45b3b171246f49df426276bc7c2c4 (127.0.103.66:35499)
I20240430 17:03:43.231669 413 webserver.cc:460] Webserver started at http://127.0.103.67:37503/ using document root <none> and password file <none>
I20240430 17:03:43.232124 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:43.232292 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:43.232514 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:43.232563 2956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46708
I20240430 17:03:43.233642 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-2-root/instance:
uuid: "5daf9d78e5324ae19f16c1394fc42b83"
format_stamp: "Formatted at 2024-04-30 17:03:43 on dist-test-slave-x21r"
I20240430 17:03:43.237857 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240430 17:03:43.240923 3169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:43.241587 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240430 17:03:43.241860 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-2-root
uuid: "5daf9d78e5324ae19f16c1394fc42b83"
format_stamp: "Formatted at 2024-04-30 17:03:43 on dist-test-slave-x21r"
I20240430 17:03:43.242120 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:43.258180 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:43.259214 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:43.260493 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:03:43.262605 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:03:43.262794 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:43.262996 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:03:43.263160 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240430 17:03:43.300549 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:46569
I20240430 17:03:43.300631 3231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.67:46569 every 8 connection(s)
I20240430 17:03:43.305126 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:43.312059 3235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:43.313542 3236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:43.314723 3238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:43.316352 413 server_base.cc:1034] running on GCE node
I20240430 17:03:43.316737 3232 heartbeater.cc:347] Connected to a master server at 127.0.103.126:34603
I20240430 17:03:43.317092 3232 heartbeater.cc:464] Registering TS with master...
I20240430 17:03:43.317181 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:03:43.317435 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:43.317601 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496623317584 us; error 0 us; skew 500 ppm
I20240430 17:03:43.317812 3232 heartbeater.cc:511] Master 127.0.103.126:34603 requested a full tablet report, sending...
I20240430 17:03:43.318157 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:43.319895 2956 ts_manager.cc:194] Registered new tserver with Master: 5daf9d78e5324ae19f16c1394fc42b83 (127.0.103.67:46569)
I20240430 17:03:43.320744 413 webserver.cc:460] Webserver started at http://127.0.103.68:46023/ using document root <none> and password file <none>
I20240430 17:03:43.321225 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:43.321405 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:43.321339 2956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46722
I20240430 17:03:43.321815 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:43.322961 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-3-root/instance:
uuid: "573241f0e800454fb6166c296fa4ba7e"
format_stamp: "Formatted at 2024-04-30 17:03:43 on dist-test-slave-x21r"
I20240430 17:03:43.326998 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240430 17:03:43.329994 3243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:43.330646 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240430 17:03:43.330906 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-3-root
uuid: "573241f0e800454fb6166c296fa4ba7e"
format_stamp: "Formatted at 2024-04-30 17:03:43 on dist-test-slave-x21r"
I20240430 17:03:43.331193 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714496547126677-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:43.344286 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:43.345516 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:43.346900 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:03:43.349114 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:03:43.349316 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:43.349535 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:03:43.349685 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:43.387737 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:42773
I20240430 17:03:43.387853 3305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.68:42773 every 8 connection(s)
I20240430 17:03:43.399997 3306 heartbeater.cc:347] Connected to a master server at 127.0.103.126:34603
I20240430 17:03:43.400326 3306 heartbeater.cc:464] Registering TS with master...
I20240430 17:03:43.401011 3306 heartbeater.cc:511] Master 127.0.103.126:34603 requested a full tablet report, sending...
I20240430 17:03:43.402595 2956 ts_manager.cc:194] Registered new tserver with Master: 573241f0e800454fb6166c296fa4ba7e (127.0.103.68:42773)
I20240430 17:03:43.402959 413 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.012323944s
I20240430 17:03:43.403848 2956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46730
I20240430 17:03:43.933087 413 test_util.cc:268] Using random seed: 1513944714
I20240430 17:03:43.953848 2956 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46752:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 17:03:43.996848 3123 tablet_service.cc:1467] Processing CreateTablet for tablet 5258622c3d3b4209a80a8424d6cffbf7 (DEFAULT_TABLE table=test-workload [id=0f6d7e638d3e4506a6708639e2ec9799]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 17:03:43.998709 3123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5258622c3d3b4209a80a8424d6cffbf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:03:44.007367 3048 tablet_service.cc:1467] Processing CreateTablet for tablet 5258622c3d3b4209a80a8424d6cffbf7 (DEFAULT_TABLE table=test-workload [id=0f6d7e638d3e4506a6708639e2ec9799]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 17:03:44.008592 3048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5258622c3d3b4209a80a8424d6cffbf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:03:44.008960 3271 tablet_service.cc:1467] Processing CreateTablet for tablet 5258622c3d3b4209a80a8424d6cffbf7 (DEFAULT_TABLE table=test-workload [id=0f6d7e638d3e4506a6708639e2ec9799]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 17:03:44.009876 3271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5258622c3d3b4209a80a8424d6cffbf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:03:44.016161 3327 tablet_bootstrap.cc:492] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4: Bootstrap starting.
I20240430 17:03:44.022132 3327 tablet_bootstrap.cc:654] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4: Neither blocks nor log segments found. Creating new log.
I20240430 17:03:44.023989 3328 tablet_bootstrap.cc:492] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac: Bootstrap starting.
I20240430 17:03:44.025887 3329 tablet_bootstrap.cc:492] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e: Bootstrap starting.
I20240430 17:03:44.032550 3328 tablet_bootstrap.cc:654] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac: Neither blocks nor log segments found. Creating new log.
I20240430 17:03:44.032923 3329 tablet_bootstrap.cc:654] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e: Neither blocks nor log segments found. Creating new log.
I20240430 17:03:44.033782 3327 tablet_bootstrap.cc:492] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4: No bootstrap required, opened a new log
I20240430 17:03:44.034160 3327 ts_tablet_manager.cc:1367] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.017s
I20240430 17:03:44.036469 3327 raft_consensus.cc:348] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d45b3b171246f49df426276bc7c2c4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35499 } } peers { permanent_uuid: "4013dd8c69644fe0b0586bd800b734ac" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45679 } } peers { permanent_uuid: "573241f0e800454fb6166c296fa4ba7e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42773 } }
I20240430 17:03:44.037356 3327 raft_consensus.cc:374] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:03:44.037686 3327 raft_consensus.cc:729] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d45b3b171246f49df426276bc7c2c4, State: Initialized, Role: FOLLOWER
I20240430 17:03:44.038486 3329 tablet_bootstrap.cc:492] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e: No bootstrap required, opened a new log
I20240430 17:03:44.038489 3327 consensus_queue.cc:260] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d45b3b171246f49df426276bc7c2c4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35499 } } peers { permanent_uuid: "4013dd8c69644fe0b0586bd800b734ac" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45679 } } peers { permanent_uuid: "573241f0e800454fb6166c296fa4ba7e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42773 } }
I20240430 17:03:44.039238 3329 ts_tablet_manager.cc:1367] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240430 17:03:44.040002 3328 tablet_bootstrap.cc:492] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac: No bootstrap required, opened a new log
I20240430 17:03:44.040522 3328 ts_tablet_manager.cc:1367] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20240430 17:03:44.041213 3158 heartbeater.cc:503] Master 127.0.103.126:34603 was elected leader, sending a full tablet report...
I20240430 17:03:44.041544 3327 ts_tablet_manager.cc:1398] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240430 17:03:44.042486 3329 raft_consensus.cc:348] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d45b3b171246f49df426276bc7c2c4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35499 } } peers { permanent_uuid: "4013dd8c69644fe0b0586bd800b734ac" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45679 } } peers { permanent_uuid: "573241f0e800454fb6166c296fa4ba7e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42773 } }
I20240430 17:03:44.043371 3329 raft_consensus.cc:374] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:03:44.043659 3329 raft_consensus.cc:729] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573241f0e800454fb6166c296fa4ba7e, State: Initialized, Role: FOLLOWER
I20240430 17:03:44.044364 3328 raft_consensus.cc:348] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d45b3b171246f49df426276bc7c2c4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35499 } } peers { permanent_uuid: "4013dd8c69644fe0b0586bd800b734ac" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45679 } } peers { permanent_uuid: "573241f0e800454fb6166c296fa4ba7e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42773 } }
I20240430 17:03:44.044409 3329 consensus_queue.cc:260] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d45b3b171246f49df426276bc7c2c4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35499 } } peers { permanent_uuid: "4013dd8c69644fe0b0586bd800b734ac" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45679 } } peers { permanent_uuid: "573241f0e800454fb6166c296fa4ba7e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42773 } }
I20240430 17:03:44.045341 3328 raft_consensus.cc:374] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:03:44.045668 3328 raft_consensus.cc:729] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4013dd8c69644fe0b0586bd800b734ac, State: Initialized, Role: FOLLOWER
I20240430 17:03:44.046562 3328 consensus_queue.cc:260] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d45b3b171246f49df426276bc7c2c4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35499 } } peers { permanent_uuid: "4013dd8c69644fe0b0586bd800b734ac" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45679 } } peers { permanent_uuid: "573241f0e800454fb6166c296fa4ba7e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42773 } }
I20240430 17:03:44.049535 3083 heartbeater.cc:503] Master 127.0.103.126:34603 was elected leader, sending a full tablet report...
I20240430 17:03:44.050372 3328 ts_tablet_manager.cc:1398] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240430 17:03:44.052001 3306 heartbeater.cc:503] Master 127.0.103.126:34603 was elected leader, sending a full tablet report...
I20240430 17:03:44.055437 3329 ts_tablet_manager.cc:1398] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e: Time spent starting tablet: real 0.015s user 0.003s sys 0.008s
W20240430 17:03:44.115303 3084 tablet.cc:2369] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 17:03:44.271981 3333 raft_consensus.cc:482] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:03:44.272495 3333 raft_consensus.cc:504] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d45b3b171246f49df426276bc7c2c4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35499 } } peers { permanent_uuid: "4013dd8c69644fe0b0586bd800b734ac" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45679 } } peers { permanent_uuid: "573241f0e800454fb6166c296fa4ba7e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42773 } }
I20240430 17:03:44.274338 3333 leader_election.cc:288] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4013dd8c69644fe0b0586bd800b734ac (127.0.103.65:45679), 573241f0e800454fb6166c296fa4ba7e (127.0.103.68:42773)
I20240430 17:03:44.284719 3281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5258622c3d3b4209a80a8424d6cffbf7" candidate_uuid: "a5d45b3b171246f49df426276bc7c2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573241f0e800454fb6166c296fa4ba7e" is_pre_election: true
I20240430 17:03:44.284823 3058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5258622c3d3b4209a80a8424d6cffbf7" candidate_uuid: "a5d45b3b171246f49df426276bc7c2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4013dd8c69644fe0b0586bd800b734ac" is_pre_election: true
I20240430 17:03:44.285454 3281 raft_consensus.cc:2454] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d45b3b171246f49df426276bc7c2c4 in term 0.
I20240430 17:03:44.285533 3058 raft_consensus.cc:2454] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d45b3b171246f49df426276bc7c2c4 in term 0.
I20240430 17:03:44.286492 3097 leader_election.cc:302] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573241f0e800454fb6166c296fa4ba7e, a5d45b3b171246f49df426276bc7c2c4; no voters:
I20240430 17:03:44.287132 3333 raft_consensus.cc:2781] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:03:44.287480 3333 raft_consensus.cc:482] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:03:44.287741 3333 raft_consensus.cc:3037] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:03:44.292424 3333 raft_consensus.cc:504] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d45b3b171246f49df426276bc7c2c4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35499 } } peers { permanent_uuid: "4013dd8c69644fe0b0586bd800b734ac" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45679 } } peers { permanent_uuid: "573241f0e800454fb6166c296fa4ba7e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42773 } }
I20240430 17:03:44.293804 3333 leader_election.cc:288] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [CANDIDATE]: Term 1 election: Requested vote from peers 4013dd8c69644fe0b0586bd800b734ac (127.0.103.65:45679), 573241f0e800454fb6166c296fa4ba7e (127.0.103.68:42773)
I20240430 17:03:44.294456 3058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5258622c3d3b4209a80a8424d6cffbf7" candidate_uuid: "a5d45b3b171246f49df426276bc7c2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4013dd8c69644fe0b0586bd800b734ac"
I20240430 17:03:44.294679 3281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5258622c3d3b4209a80a8424d6cffbf7" candidate_uuid: "a5d45b3b171246f49df426276bc7c2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573241f0e800454fb6166c296fa4ba7e"
I20240430 17:03:44.294889 3058 raft_consensus.cc:3037] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:03:44.295207 3281 raft_consensus.cc:3037] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:03:44.299973 3058 raft_consensus.cc:2454] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d45b3b171246f49df426276bc7c2c4 in term 1.
I20240430 17:03:44.300041 3281 raft_consensus.cc:2454] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d45b3b171246f49df426276bc7c2c4 in term 1.
I20240430 17:03:44.300971 3096 leader_election.cc:302] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4013dd8c69644fe0b0586bd800b734ac, a5d45b3b171246f49df426276bc7c2c4; no voters:
I20240430 17:03:44.301597 3333 raft_consensus.cc:2781] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:03:44.302589 3333 raft_consensus.cc:686] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 1 LEADER]: Becoming Leader. State: Replica: a5d45b3b171246f49df426276bc7c2c4, State: Running, Role: LEADER
I20240430 17:03:44.303260 3333 consensus_queue.cc:237] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d45b3b171246f49df426276bc7c2c4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35499 } } peers { permanent_uuid: "4013dd8c69644fe0b0586bd800b734ac" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45679 } } peers { permanent_uuid: "573241f0e800454fb6166c296fa4ba7e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42773 } }
I20240430 17:03:44.310731 2955 catalog_manager.cc:5484] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d45b3b171246f49df426276bc7c2c4 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a5d45b3b171246f49df426276bc7c2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d45b3b171246f49df426276bc7c2c4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4013dd8c69644fe0b0586bd800b734ac" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "573241f0e800454fb6166c296fa4ba7e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42773 } health_report { overall_health: UNKNOWN } } }
I20240430 17:03:44.323757 3232 heartbeater.cc:503] Master 127.0.103.126:34603 was elected leader, sending a full tablet report...
I20240430 17:03:44.708293 3333 consensus_queue.cc:1035] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4013dd8c69644fe0b0586bd800b734ac" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 17:03:44.720500 3333 consensus_queue.cc:1035] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "573241f0e800454fb6166c296fa4ba7e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:03:44.893112 3009 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:44.894589 3009 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:44.894912 3009 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:46.895829 3009 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:46.897179 3009 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:46.897475 3009 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:48.898500 3009 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:48.899942 3009 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:48.900211 3009 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:50.901095 3009 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:50.902369 3009 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:50.902645 3009 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:52.903527 3009 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:52.904753 3009 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:52.905037 3009 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:54.905855 3009 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:54.907074 3009 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:54.907385 3009 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:55.439453 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20240430 17:03:55.465456 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:03:55.466086 413 tablet_replica.cc:330] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac: stopping tablet replica
I20240430 17:03:55.466614 413 raft_consensus.cc:2229] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:03:55.467093 413 raft_consensus.cc:2258] T 5258622c3d3b4209a80a8424d6cffbf7 P 4013dd8c69644fe0b0586bd800b734ac [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 17:03:55.475481 3096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:45679: connect: Connection refused (error 111)
W20240430 17:03:55.478590 3096 consensus_peers.cc:485] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 -> Peer 4013dd8c69644fe0b0586bd800b734ac (127.0.103.65:45679): Couldn't send request to peer 4013dd8c69644fe0b0586bd800b734ac. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:45679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 17:03:55.486172 413 tablet_server.cc:194] TabletServer@127.0.103.65:0 shutdown complete.
I20240430 17:03:55.496438 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20240430 17:03:55.516997 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:03:55.517616 413 tablet_replica.cc:330] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4: stopping tablet replica
I20240430 17:03:55.518102 413 raft_consensus.cc:2229] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:03:55.518890 413 raft_consensus.cc:2258] T 5258622c3d3b4209a80a8424d6cffbf7 P a5d45b3b171246f49df426276bc7c2c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:03:55.536959 413 tablet_server.cc:194] TabletServer@127.0.103.66:0 shutdown complete.
I20240430 17:03:55.546993 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20240430 17:03:55.569705 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:03:55.584985 413 tablet_server.cc:194] TabletServer@127.0.103.67:0 shutdown complete.
I20240430 17:03:55.592978 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20240430 17:03:55.612774 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:03:55.613375 413 tablet_replica.cc:330] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e: stopping tablet replica
I20240430 17:03:55.613901 413 raft_consensus.cc:2229] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:03:55.614311 413 raft_consensus.cc:2258] T 5258622c3d3b4209a80a8424d6cffbf7 P 573241f0e800454fb6166c296fa4ba7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:03:55.632778 413 tablet_server.cc:194] TabletServer@127.0.103.68:0 shutdown complete.
I20240430 17:03:55.642233 413 master.cc:537] Master@127.0.103.126:34603 shutting down...
I20240430 17:03:55.659682 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:03:55.660288 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:03:55.660601 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P 92c52a11991d4a21bb9059571adc5dae: stopping tablet replica
I20240430 17:03:55.677973 413 master.cc:552] Master@127.0.103.126:34603 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12938 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240430 17:03:55.710518 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:37253
I20240430 17:03:55.711639 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:55.717895 3353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:55.718708 3354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:55.719715 3356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:55.720402 413 server_base.cc:1034] running on GCE node
I20240430 17:03:55.721190 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:03:55.721375 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:55.721527 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496635721510 us; error 0 us; skew 500 ppm
I20240430 17:03:55.722021 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:55.724397 413 webserver.cc:460] Webserver started at http://127.0.103.126:36105/ using document root <none> and password file <none>
I20240430 17:03:55.724865 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:55.725046 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:55.725297 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:55.726500 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/master-0-root/instance:
uuid: "f4e66e74fb2a45648c61078bf82310f9"
format_stamp: "Formatted at 2024-04-30 17:03:55 on dist-test-slave-x21r"
I20240430 17:03:55.730923 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240430 17:03:55.733904 3361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:55.734630 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240430 17:03:55.734848 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/master-0-root
uuid: "f4e66e74fb2a45648c61078bf82310f9"
format_stamp: "Formatted at 2024-04-30 17:03:55 on dist-test-slave-x21r"
I20240430 17:03:55.735090 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:55.750058 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:55.751132 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:55.786736 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:37253
I20240430 17:03:55.786828 3412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.126:37253 every 8 connection(s)
I20240430 17:03:55.790611 3413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:03:55.800479 3413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9: Bootstrap starting.
I20240430 17:03:55.804404 3413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9: Neither blocks nor log segments found. Creating new log.
I20240430 17:03:55.808096 3413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9: No bootstrap required, opened a new log
I20240430 17:03:55.809900 3413 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e66e74fb2a45648c61078bf82310f9" member_type: VOTER }
I20240430 17:03:55.810357 3413 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:03:55.810572 3413 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e66e74fb2a45648c61078bf82310f9, State: Initialized, Role: FOLLOWER
I20240430 17:03:55.811146 3413 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e66e74fb2a45648c61078bf82310f9" member_type: VOTER }
I20240430 17:03:55.811601 3413 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:03:55.811820 3413 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:03:55.812078 3413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:03:55.815848 3413 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e66e74fb2a45648c61078bf82310f9" member_type: VOTER }
I20240430 17:03:55.816306 3413 leader_election.cc:302] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4e66e74fb2a45648c61078bf82310f9; no voters:
I20240430 17:03:55.817422 3413 leader_election.cc:288] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:03:55.817818 3416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:03:55.818997 3416 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [term 1 LEADER]: Becoming Leader. State: Replica: f4e66e74fb2a45648c61078bf82310f9, State: Running, Role: LEADER
I20240430 17:03:55.819576 3416 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e66e74fb2a45648c61078bf82310f9" member_type: VOTER }
I20240430 17:03:55.820144 3413 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 17:03:55.822505 3418 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4e66e74fb2a45648c61078bf82310f9. Latest consensus state: current_term: 1 leader_uuid: "f4e66e74fb2a45648c61078bf82310f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e66e74fb2a45648c61078bf82310f9" member_type: VOTER } }
I20240430 17:03:55.822397 3417 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4e66e74fb2a45648c61078bf82310f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e66e74fb2a45648c61078bf82310f9" member_type: VOTER } }
I20240430 17:03:55.823261 3418 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [sys.catalog]: This master's current role is: LEADER
I20240430 17:03:55.823348 3417 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [sys.catalog]: This master's current role is: LEADER
I20240430 17:03:55.826118 3422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 17:03:55.833837 3422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 17:03:55.834710 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240430 17:03:55.841804 3422 catalog_manager.cc:1347] Generated new cluster ID: e739b391e9b14c0294aead744aa32c98
I20240430 17:03:55.842044 3422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 17:03:55.872992 3422 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 17:03:55.874158 3422 catalog_manager.cc:1504] Loading token signing keys...
I20240430 17:03:55.888968 3422 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9: Generated new TSK 0
I20240430 17:03:55.889497 3422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 17:03:55.900840 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:55.906661 3434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:55.907886 3435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:55.908569 3437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:55.909509 413 server_base.cc:1034] running on GCE node
I20240430 17:03:55.910180 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:03:55.910362 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:55.910509 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496635910493 us; error 0 us; skew 500 ppm
I20240430 17:03:55.910995 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:55.913122 413 webserver.cc:460] Webserver started at http://127.0.103.65:35993/ using document root <none> and password file <none>
I20240430 17:03:55.913550 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:55.913713 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:55.913945 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:55.914928 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-0-root/instance:
uuid: "8e59d21917ff4113b9aa9a0dfb002a3e"
format_stamp: "Formatted at 2024-04-30 17:03:55 on dist-test-slave-x21r"
I20240430 17:03:55.918948 413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240430 17:03:55.921973 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:55.922637 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240430 17:03:55.922885 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-0-root
uuid: "8e59d21917ff4113b9aa9a0dfb002a3e"
format_stamp: "Formatted at 2024-04-30 17:03:55 on dist-test-slave-x21r"
I20240430 17:03:55.923144 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:55.936282 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:55.938081 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:55.939400 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:03:55.941493 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:03:55.941679 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:55.941898 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:03:55.942054 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:55.978659 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39151
I20240430 17:03:55.978742 3504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.65:39151 every 8 connection(s)
I20240430 17:03:55.982910 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:55.991073 3509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:55.992143 3510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:55.993710 3512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:55.995209 413 server_base.cc:1034] running on GCE node
I20240430 17:03:55.996162 3505 heartbeater.cc:347] Connected to a master server at 127.0.103.126:37253
I20240430 17:03:55.996240 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240430 17:03:55.996485 3505 heartbeater.cc:464] Registering TS with master...
W20240430 17:03:55.996496 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:55.996950 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496635996933 us; error 0 us; skew 500 ppm
I20240430 17:03:55.997318 3505 heartbeater.cc:511] Master 127.0.103.126:37253 requested a full tablet report, sending...
I20240430 17:03:55.997498 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:55.999521 3378 ts_manager.cc:194] Registered new tserver with Master: 8e59d21917ff4113b9aa9a0dfb002a3e (127.0.103.65:39151)
I20240430 17:03:56.000305 413 webserver.cc:460] Webserver started at http://127.0.103.66:32799/ using document root <none> and password file <none>
I20240430 17:03:56.000901 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:56.001147 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:56.001441 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:56.001961 3378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57690
I20240430 17:03:56.002981 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-1-root/instance:
uuid: "a88c154934b84d5eacbbcabe1b8c6ef4"
format_stamp: "Formatted at 2024-04-30 17:03:55 on dist-test-slave-x21r"
I20240430 17:03:56.007043 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240430 17:03:56.010080 3517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:56.010782 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240430 17:03:56.011035 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-1-root
uuid: "a88c154934b84d5eacbbcabe1b8c6ef4"
format_stamp: "Formatted at 2024-04-30 17:03:55 on dist-test-slave-x21r"
I20240430 17:03:56.011337 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:56.028772 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:56.029901 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:56.031210 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:03:56.033380 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:03:56.033562 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:56.033767 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:03:56.033924 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:56.070219 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:34333
I20240430 17:03:56.070307 3579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.66:34333 every 8 connection(s)
I20240430 17:03:56.074589 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:03:56.081715 3583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:03:56.084744 3584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:56.085392 3580 heartbeater.cc:347] Connected to a master server at 127.0.103.126:37253
W20240430 17:03:56.085603 3586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:03:56.085709 3580 heartbeater.cc:464] Registering TS with master...
I20240430 17:03:56.086560 413 server_base.cc:1034] running on GCE node
I20240430 17:03:56.086639 3580 heartbeater.cc:511] Master 127.0.103.126:37253 requested a full tablet report, sending...
I20240430 17:03:56.087301 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:03:56.087509 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:03:56.087671 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496636087649 us; error 0 us; skew 500 ppm
I20240430 17:03:56.088303 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:03:56.088565 3378 ts_manager.cc:194] Registered new tserver with Master: a88c154934b84d5eacbbcabe1b8c6ef4 (127.0.103.66:34333)
I20240430 17:03:56.090122 3378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57696
I20240430 17:03:56.090857 413 webserver.cc:460] Webserver started at http://127.0.103.67:40069/ using document root <none> and password file <none>
I20240430 17:03:56.091446 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:03:56.091635 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:03:56.091871 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:03:56.092945 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-2-root/instance:
uuid: "5423a1a082da415e82e05cbcf6078de3"
format_stamp: "Formatted at 2024-04-30 17:03:56 on dist-test-slave-x21r"
I20240430 17:03:56.097030 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240430 17:03:56.099906 3591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:56.100581 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240430 17:03:56.100898 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-2-root
uuid: "5423a1a082da415e82e05cbcf6078de3"
format_stamp: "Formatted at 2024-04-30 17:03:56 on dist-test-slave-x21r"
I20240430 17:03:56.101167 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714496547126677-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:03:56.139684 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:03:56.140754 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:03:56.142131 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:03:56.144223 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:03:56.144421 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:56.144634 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:03:56.144783 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:03:56.180400 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:39173
I20240430 17:03:56.180497 3653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.67:39173 every 8 connection(s)
I20240430 17:03:56.192757 3654 heartbeater.cc:347] Connected to a master server at 127.0.103.126:37253
I20240430 17:03:56.193137 3654 heartbeater.cc:464] Registering TS with master...
I20240430 17:03:56.193749 3654 heartbeater.cc:511] Master 127.0.103.126:37253 requested a full tablet report, sending...
I20240430 17:03:56.195497 3378 ts_manager.cc:194] Registered new tserver with Master: 5423a1a082da415e82e05cbcf6078de3 (127.0.103.67:39173)
I20240430 17:03:56.196504 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013128842s
I20240430 17:03:56.196681 3378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57708
I20240430 17:03:57.004415 3505 heartbeater.cc:503] Master 127.0.103.126:37253 was elected leader, sending a full tablet report...
I20240430 17:03:57.092358 3580 heartbeater.cc:503] Master 127.0.103.126:37253 was elected leader, sending a full tablet report...
I20240430 17:03:57.198900 3654 heartbeater.cc:503] Master 127.0.103.126:37253 was elected leader, sending a full tablet report...
I20240430 17:03:57.238894 413 test_util.cc:268] Using random seed: 1527250523
I20240430 17:03:57.264267 3378 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57716:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 17:03:57.266088 3378 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 17:03:57.323390 3470 tablet_service.cc:1467] Processing CreateTablet for tablet 731906cb373c4e5da6adcee4f4041dd3 (DEFAULT_TABLE table=test-workload [id=b1443ddccca6453e98536f1632e0c502]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 17:03:57.324088 3619 tablet_service.cc:1467] Processing CreateTablet for tablet 731906cb373c4e5da6adcee4f4041dd3 (DEFAULT_TABLE table=test-workload [id=b1443ddccca6453e98536f1632e0c502]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 17:03:57.324748 3470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 731906cb373c4e5da6adcee4f4041dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:03:57.325301 3619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 731906cb373c4e5da6adcee4f4041dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:03:57.327909 3545 tablet_service.cc:1467] Processing CreateTablet for tablet 731906cb373c4e5da6adcee4f4041dd3 (DEFAULT_TABLE table=test-workload [id=b1443ddccca6453e98536f1632e0c502]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 17:03:57.329133 3545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 731906cb373c4e5da6adcee4f4041dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:03:57.342109 3676 tablet_bootstrap.cc:492] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3: Bootstrap starting.
I20240430 17:03:57.343295 3675 tablet_bootstrap.cc:492] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e: Bootstrap starting.
I20240430 17:03:57.344810 3677 tablet_bootstrap.cc:492] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4: Bootstrap starting.
I20240430 17:03:57.348349 3676 tablet_bootstrap.cc:654] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3: Neither blocks nor log segments found. Creating new log.
I20240430 17:03:57.348775 3675 tablet_bootstrap.cc:654] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e: Neither blocks nor log segments found. Creating new log.
I20240430 17:03:57.350745 3677 tablet_bootstrap.cc:654] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4: Neither blocks nor log segments found. Creating new log.
I20240430 17:03:57.353222 3675 tablet_bootstrap.cc:492] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e: No bootstrap required, opened a new log
I20240430 17:03:57.353578 3675 ts_tablet_manager.cc:1367] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240430 17:03:57.354312 3676 tablet_bootstrap.cc:492] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3: No bootstrap required, opened a new log
I20240430 17:03:57.354679 3676 ts_tablet_manager.cc:1367] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20240430 17:03:57.356113 3677 tablet_bootstrap.cc:492] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4: No bootstrap required, opened a new log
I20240430 17:03:57.355825 3675 raft_consensus.cc:348] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34333 } } peers { permanent_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39151 } } peers { permanent_uuid: "5423a1a082da415e82e05cbcf6078de3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39173 } }
I20240430 17:03:57.356603 3677 ts_tablet_manager.cc:1367] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.002s
I20240430 17:03:57.356709 3675 raft_consensus.cc:374] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:03:57.357182 3675 raft_consensus.cc:729] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e59d21917ff4113b9aa9a0dfb002a3e, State: Initialized, Role: FOLLOWER
I20240430 17:03:57.357208 3676 raft_consensus.cc:348] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34333 } } peers { permanent_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39151 } } peers { permanent_uuid: "5423a1a082da415e82e05cbcf6078de3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39173 } }
I20240430 17:03:57.358191 3676 raft_consensus.cc:374] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:03:57.358173 3675 consensus_queue.cc:260] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34333 } } peers { permanent_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39151 } } peers { permanent_uuid: "5423a1a082da415e82e05cbcf6078de3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39173 } }
I20240430 17:03:57.358714 3676 raft_consensus.cc:729] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5423a1a082da415e82e05cbcf6078de3, State: Initialized, Role: FOLLOWER
I20240430 17:03:57.359246 3677 raft_consensus.cc:348] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34333 } } peers { permanent_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39151 } } peers { permanent_uuid: "5423a1a082da415e82e05cbcf6078de3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39173 } }
I20240430 17:03:57.359493 3676 consensus_queue.cc:260] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34333 } } peers { permanent_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39151 } } peers { permanent_uuid: "5423a1a082da415e82e05cbcf6078de3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39173 } }
I20240430 17:03:57.360018 3677 raft_consensus.cc:374] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:03:57.360368 3677 raft_consensus.cc:729] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88c154934b84d5eacbbcabe1b8c6ef4, State: Initialized, Role: FOLLOWER
I20240430 17:03:57.361119 3675 ts_tablet_manager.cc:1398] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240430 17:03:57.361160 3677 consensus_queue.cc:260] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34333 } } peers { permanent_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39151 } } peers { permanent_uuid: "5423a1a082da415e82e05cbcf6078de3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39173 } }
I20240430 17:03:57.364931 3677 ts_tablet_manager.cc:1398] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20240430 17:03:57.368177 3676 ts_tablet_manager.cc:1398] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20240430 17:03:57.746884 3682 raft_consensus.cc:482] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:03:57.747650 3682 raft_consensus.cc:504] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34333 } } peers { permanent_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39151 } } peers { permanent_uuid: "5423a1a082da415e82e05cbcf6078de3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39173 } }
I20240430 17:03:57.749991 3682 leader_election.cc:288] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a88c154934b84d5eacbbcabe1b8c6ef4 (127.0.103.66:34333), 8e59d21917ff4113b9aa9a0dfb002a3e (127.0.103.65:39151)
I20240430 17:03:57.756251 3681 raft_consensus.cc:482] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:03:57.756987 3681 raft_consensus.cc:504] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34333 } } peers { permanent_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39151 } } peers { permanent_uuid: "5423a1a082da415e82e05cbcf6078de3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39173 } }
I20240430 17:03:57.761116 3681 leader_election.cc:288] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a88c154934b84d5eacbbcabe1b8c6ef4 (127.0.103.66:34333), 5423a1a082da415e82e05cbcf6078de3 (127.0.103.67:39173)
I20240430 17:03:57.766275 3480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "731906cb373c4e5da6adcee4f4041dd3" candidate_uuid: "5423a1a082da415e82e05cbcf6078de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" is_pre_election: true
I20240430 17:03:57.767513 3480 raft_consensus.cc:2454] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5423a1a082da415e82e05cbcf6078de3 in term 0.
I20240430 17:03:57.769163 3595 leader_election.cc:302] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5423a1a082da415e82e05cbcf6078de3, 8e59d21917ff4113b9aa9a0dfb002a3e; no voters:
I20240430 17:03:57.770316 3682 raft_consensus.cc:2781] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:03:57.770809 3682 raft_consensus.cc:482] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:03:57.771275 3682 raft_consensus.cc:3037] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:03:57.772915 3555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "731906cb373c4e5da6adcee4f4041dd3" candidate_uuid: "5423a1a082da415e82e05cbcf6078de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" is_pre_election: true
I20240430 17:03:57.774057 3555 raft_consensus.cc:2454] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5423a1a082da415e82e05cbcf6078de3 in term 0.
I20240430 17:03:57.781164 3682 raft_consensus.cc:504] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34333 } } peers { permanent_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39151 } } peers { permanent_uuid: "5423a1a082da415e82e05cbcf6078de3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39173 } }
I20240430 17:03:57.782749 3555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "731906cb373c4e5da6adcee4f4041dd3" candidate_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" is_pre_election: true
I20240430 17:03:57.783713 3682 leader_election.cc:288] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [CANDIDATE]: Term 1 election: Requested vote from peers a88c154934b84d5eacbbcabe1b8c6ef4 (127.0.103.66:34333), 8e59d21917ff4113b9aa9a0dfb002a3e (127.0.103.65:39151)
I20240430 17:03:57.784154 3480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "731906cb373c4e5da6adcee4f4041dd3" candidate_uuid: "5423a1a082da415e82e05cbcf6078de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e"
I20240430 17:03:57.783443 3555 raft_consensus.cc:2454] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e59d21917ff4113b9aa9a0dfb002a3e in term 0.
I20240430 17:03:57.784927 3480 raft_consensus.cc:3037] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:03:57.784152 3554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "731906cb373c4e5da6adcee4f4041dd3" candidate_uuid: "5423a1a082da415e82e05cbcf6078de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4"
I20240430 17:03:57.786232 3444 leader_election.cc:302] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e59d21917ff4113b9aa9a0dfb002a3e, a88c154934b84d5eacbbcabe1b8c6ef4; no voters:
I20240430 17:03:57.786281 3554 raft_consensus.cc:3037] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:03:57.792807 3480 raft_consensus.cc:2454] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5423a1a082da415e82e05cbcf6078de3 in term 1.
I20240430 17:03:57.792853 3629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "731906cb373c4e5da6adcee4f4041dd3" candidate_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5423a1a082da415e82e05cbcf6078de3" is_pre_election: true
I20240430 17:03:57.793269 3681 raft_consensus.cc:2741] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240430 17:03:57.793702 3629 raft_consensus.cc:2379] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e59d21917ff4113b9aa9a0dfb002a3e in current term 1: Already voted for candidate 5423a1a082da415e82e05cbcf6078de3 in this term.
I20240430 17:03:57.793992 3595 leader_election.cc:302] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5423a1a082da415e82e05cbcf6078de3, 8e59d21917ff4113b9aa9a0dfb002a3e; no voters:
I20240430 17:03:57.795552 3554 raft_consensus.cc:2454] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5423a1a082da415e82e05cbcf6078de3 in term 1.
I20240430 17:03:57.796284 3682 raft_consensus.cc:2781] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:03:57.797582 3682 raft_consensus.cc:686] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 1 LEADER]: Becoming Leader. State: Replica: 5423a1a082da415e82e05cbcf6078de3, State: Running, Role: LEADER
I20240430 17:03:57.798610 3682 consensus_queue.cc:237] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34333 } } peers { permanent_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39151 } } peers { permanent_uuid: "5423a1a082da415e82e05cbcf6078de3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39173 } }
I20240430 17:03:57.805742 3378 catalog_manager.cc:5484] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5423a1a082da415e82e05cbcf6078de3 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "5423a1a082da415e82e05cbcf6078de3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5423a1a082da415e82e05cbcf6078de3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39173 } health_report { overall_health: HEALTHY } } }
I20240430 17:03:57.832469 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:03:57.834283 3431 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:57.835729 3431 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:57.836066 3431 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:58.330893 3695 consensus_queue.cc:1035] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e59d21917ff4113b9aa9a0dfb002a3e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:03:58.344178 3701 consensus_queue.cc:1035] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88c154934b84d5eacbbcabe1b8c6ef4" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:03:58.840744 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:03:59.836931 3431 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:03:59.838141 3431 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:03:59.838426 3431 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:03:59.848268 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:04:00.855474 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:04:01.839386 3431 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:04:01.840621 3431 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:04:01.840888 3431 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:04:01.864651 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:04:02.873188 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:04:03.841734 3431 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:04:03.842962 3431 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:04:03.843298 3431 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:04:03.880530 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:04:04.887677 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:04:05.844259 3431 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:04:05.845484 3431 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:04:05.845777 3431 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:04:05.895335 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:04:06.901731 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:04:07.847002 3431 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:04:07.848747 3431 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:04:07.849062 3431 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:04:07.908322 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:04:08.914407 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:04:09.849949 3431 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:04:09.851230 3431 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:04:09.851539 3431 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:04:09.920317 3426 placement_policy_util.cc:407] tablet 731906cb373c4e5da6adcee4f4041dd3: detected majority of replicas (2 of 3) at location L0
I20240430 17:04:09.966668 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20240430 17:04:09.991034 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:04:09.991852 413 tablet_replica.cc:330] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e: stopping tablet replica
I20240430 17:04:09.992451 413 raft_consensus.cc:2229] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:04:09.993062 413 raft_consensus.cc:2258] T 731906cb373c4e5da6adcee4f4041dd3 P 8e59d21917ff4113b9aa9a0dfb002a3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:10.014461 413 tablet_server.cc:194] TabletServer@127.0.103.65:0 shutdown complete.
I20240430 17:04:10.026257 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20240430 17:04:10.046396 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:04:10.047055 413 tablet_replica.cc:330] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4: stopping tablet replica
I20240430 17:04:10.047650 413 raft_consensus.cc:2229] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:04:10.048149 413 raft_consensus.cc:2258] T 731906cb373c4e5da6adcee4f4041dd3 P a88c154934b84d5eacbbcabe1b8c6ef4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:10.067065 413 tablet_server.cc:194] TabletServer@127.0.103.66:0 shutdown complete.
I20240430 17:04:10.077860 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20240430 17:04:10.100476 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:04:10.101219 413 tablet_replica.cc:330] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3: stopping tablet replica
W20240430 17:04:10.101447 3593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.66:34333: connect: Connection refused (error 111)
I20240430 17:04:10.102131 413 raft_consensus.cc:2229] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:04:10.103490 413 raft_consensus.cc:2258] T 731906cb373c4e5da6adcee4f4041dd3 P 5423a1a082da415e82e05cbcf6078de3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:10.122785 413 tablet_server.cc:194] TabletServer@127.0.103.67:0 shutdown complete.
I20240430 17:04:10.132985 413 master.cc:537] Master@127.0.103.126:37253 shutting down...
I20240430 17:04:10.151885 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:04:10.152554 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:10.152876 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P f4e66e74fb2a45648c61078bf82310f9: stopping tablet replica
I20240430 17:04:10.170545 413 master.cc:552] Master@127.0.103.126:37253 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (14493 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
/home/jenkins-slave/workspace/build_and_test_flaky@3/src/kudu/master/auto_rebalancer-test.cc:516: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] AutoRebalancerTest.TestMaxMovesPerServer (6 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240430 17:04:10.210294 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:37467
I20240430 17:04:10.211375 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:04:10.217613 3711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:04:10.218322 3712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:04:10.219085 3714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:04:10.219756 413 server_base.cc:1034] running on GCE node
I20240430 17:04:10.220487 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:04:10.220681 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:04:10.220829 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496650220812 us; error 0 us; skew 500 ppm
I20240430 17:04:10.221285 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:04:10.227363 413 webserver.cc:460] Webserver started at http://127.0.103.126:45341/ using document root <none> and password file <none>
I20240430 17:04:10.227839 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:04:10.228010 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:04:10.228268 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:04:10.229442 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/master-0-root/instance:
uuid: "36c59c6b68124a2cab9bdbd105645dd1"
format_stamp: "Formatted at 2024-04-30 17:04:10 on dist-test-slave-x21r"
I20240430 17:04:10.233780 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240430 17:04:10.236953 3719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:10.237654 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240430 17:04:10.237905 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/master-0-root
uuid: "36c59c6b68124a2cab9bdbd105645dd1"
format_stamp: "Formatted at 2024-04-30 17:04:10 on dist-test-slave-x21r"
I20240430 17:04:10.238168 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:04:10.267841 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:04:10.269039 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:04:10.306521 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:37467
I20240430 17:04:10.306614 3770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.126:37467 every 8 connection(s)
I20240430 17:04:10.310622 3771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:04:10.321084 3771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1: Bootstrap starting.
I20240430 17:04:10.325106 3771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1: Neither blocks nor log segments found. Creating new log.
I20240430 17:04:10.328984 3771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1: No bootstrap required, opened a new log
I20240430 17:04:10.330865 3771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c59c6b68124a2cab9bdbd105645dd1" member_type: VOTER }
I20240430 17:04:10.331373 3771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:04:10.331610 3771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c59c6b68124a2cab9bdbd105645dd1, State: Initialized, Role: FOLLOWER
I20240430 17:04:10.332211 3771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c59c6b68124a2cab9bdbd105645dd1" member_type: VOTER }
I20240430 17:04:10.332679 3771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240430 17:04:10.332885 3771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240430 17:04:10.333144 3771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:04:10.337257 3771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c59c6b68124a2cab9bdbd105645dd1" member_type: VOTER }
I20240430 17:04:10.337738 3771 leader_election.cc:302] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36c59c6b68124a2cab9bdbd105645dd1; no voters:
I20240430 17:04:10.338842 3771 leader_election.cc:288] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240430 17:04:10.339213 3774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:04:10.340530 3774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [term 1 LEADER]: Becoming Leader. State: Replica: 36c59c6b68124a2cab9bdbd105645dd1, State: Running, Role: LEADER
I20240430 17:04:10.341104 3774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c59c6b68124a2cab9bdbd105645dd1" member_type: VOTER }
I20240430 17:04:10.341807 3771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 17:04:10.343858 3775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36c59c6b68124a2cab9bdbd105645dd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c59c6b68124a2cab9bdbd105645dd1" member_type: VOTER } }
I20240430 17:04:10.343878 3776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36c59c6b68124a2cab9bdbd105645dd1. Latest consensus state: current_term: 1 leader_uuid: "36c59c6b68124a2cab9bdbd105645dd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c59c6b68124a2cab9bdbd105645dd1" member_type: VOTER } }
I20240430 17:04:10.344647 3775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [sys.catalog]: This master's current role is: LEADER
I20240430 17:04:10.344774 3776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [sys.catalog]: This master's current role is: LEADER
I20240430 17:04:10.347599 3779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 17:04:10.352291 3779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 17:04:10.356087 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240430 17:04:10.360795 3779 catalog_manager.cc:1347] Generated new cluster ID: 355d2a7a44ae40f08e72b48e11c02c19
I20240430 17:04:10.361058 3779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 17:04:10.393990 3779 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 17:04:10.395231 3779 catalog_manager.cc:1504] Loading token signing keys...
I20240430 17:04:10.415889 3779 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1: Generated new TSK 0
I20240430 17:04:10.416461 3779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 17:04:10.422183 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:04:10.428187 3792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:04:10.429370 3793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:04:10.430167 3795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:04:10.431360 413 server_base.cc:1034] running on GCE node
I20240430 17:04:10.432096 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:04:10.432287 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:04:10.432437 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496650432422 us; error 0 us; skew 500 ppm
I20240430 17:04:10.432927 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:04:10.435096 413 webserver.cc:460] Webserver started at http://127.0.103.65:35751/ using document root <none> and password file <none>
I20240430 17:04:10.435590 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:04:10.435766 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:04:10.436009 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:04:10.437050 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-0-root/instance:
uuid: "f1488849a12e4e3dae956cd218f4ed22"
format_stamp: "Formatted at 2024-04-30 17:04:10 on dist-test-slave-x21r"
I20240430 17:04:10.441418 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240430 17:04:10.444547 3800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:10.445217 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240430 17:04:10.445475 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-0-root
uuid: "f1488849a12e4e3dae956cd218f4ed22"
format_stamp: "Formatted at 2024-04-30 17:04:10 on dist-test-slave-x21r"
I20240430 17:04:10.445741 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:04:10.454378 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:04:10.455473 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:04:10.456799 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:04:10.458981 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:04:10.459192 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:10.459399 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:04:10.459555 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:10.496845 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38761
I20240430 17:04:10.496920 3862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.65:38761 every 8 connection(s)
I20240430 17:04:10.501505 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:04:10.508667 3867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:04:10.510020 3868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:04:10.511811 3870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:04:10.513813 413 server_base.cc:1034] running on GCE node
I20240430 17:04:10.514228 3863 heartbeater.cc:347] Connected to a master server at 127.0.103.126:37467
I20240430 17:04:10.514552 3863 heartbeater.cc:464] Registering TS with master...
I20240430 17:04:10.514748 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:04:10.514981 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:04:10.515156 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496650515138 us; error 0 us; skew 500 ppm
I20240430 17:04:10.515260 3863 heartbeater.cc:511] Master 127.0.103.126:37467 requested a full tablet report, sending...
I20240430 17:04:10.515718 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:04:10.517382 3736 ts_manager.cc:194] Registered new tserver with Master: f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761)
I20240430 17:04:10.518390 413 webserver.cc:460] Webserver started at http://127.0.103.66:36929/ using document root <none> and password file <none>
I20240430 17:04:10.518846 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:04:10.519028 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:04:10.519147 3736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40822
I20240430 17:04:10.519315 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:04:10.520630 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-1-root/instance:
uuid: "5e8e00d436a14372a8280644f779828f"
format_stamp: "Formatted at 2024-04-30 17:04:10 on dist-test-slave-x21r"
I20240430 17:04:10.524771 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240430 17:04:10.527824 3875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:10.528527 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240430 17:04:10.528780 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-1-root
uuid: "5e8e00d436a14372a8280644f779828f"
format_stamp: "Formatted at 2024-04-30 17:04:10 on dist-test-slave-x21r"
I20240430 17:04:10.529048 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:04:10.551548 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:04:10.552667 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:04:10.553966 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:04:10.556161 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:04:10.556349 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:10.556571 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:04:10.556717 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:10.592895 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:36435
I20240430 17:04:10.592980 3937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.66:36435 every 8 connection(s)
I20240430 17:04:10.597364 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:04:10.604319 3941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:04:10.606482 3942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:04:10.607461 3944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:04:10.609450 413 server_base.cc:1034] running on GCE node
I20240430 17:04:10.610147 3938 heartbeater.cc:347] Connected to a master server at 127.0.103.126:37467
I20240430 17:04:10.610258 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240430 17:04:10.610500 3938 heartbeater.cc:464] Registering TS with master...
W20240430 17:04:10.610535 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:04:10.610888 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496650610871 us; error 0 us; skew 500 ppm
I20240430 17:04:10.611372 3938 heartbeater.cc:511] Master 127.0.103.126:37467 requested a full tablet report, sending...
I20240430 17:04:10.611469 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:04:10.613390 3736 ts_manager.cc:194] Registered new tserver with Master: 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435)
I20240430 17:04:10.614614 3736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40824
I20240430 17:04:10.615674 413 webserver.cc:460] Webserver started at http://127.0.103.67:36105/ using document root <none> and password file <none>
I20240430 17:04:10.616137 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:04:10.616317 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:04:10.616569 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:04:10.617740 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-2-root/instance:
uuid: "64b1d8e536e141a89a54b559d8a4a5f9"
format_stamp: "Formatted at 2024-04-30 17:04:10 on dist-test-slave-x21r"
I20240430 17:04:10.622045 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240430 17:04:10.625236 3949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:10.626022 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240430 17:04:10.626253 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-2-root
uuid: "64b1d8e536e141a89a54b559d8a4a5f9"
format_stamp: "Formatted at 2024-04-30 17:04:10 on dist-test-slave-x21r"
I20240430 17:04:10.626492 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:04:10.637573 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:04:10.638577 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:04:10.640043 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:04:10.642151 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:04:10.642352 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:10.642575 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:04:10.642719 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:10.679795 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:43323
I20240430 17:04:10.679893 4011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.67:43323 every 8 connection(s)
I20240430 17:04:10.692111 4012 heartbeater.cc:347] Connected to a master server at 127.0.103.126:37467
I20240430 17:04:10.692476 4012 heartbeater.cc:464] Registering TS with master...
I20240430 17:04:10.693176 4012 heartbeater.cc:511] Master 127.0.103.126:37467 requested a full tablet report, sending...
I20240430 17:04:10.694903 3736 ts_manager.cc:194] Registered new tserver with Master: 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67:43323)
I20240430 17:04:10.695896 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013158532s
I20240430 17:04:10.696291 3736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40826
I20240430 17:04:11.521530 3863 heartbeater.cc:503] Master 127.0.103.126:37467 was elected leader, sending a full tablet report...
I20240430 17:04:11.616931 3938 heartbeater.cc:503] Master 127.0.103.126:37467 was elected leader, sending a full tablet report...
I20240430 17:04:11.698588 4012 heartbeater.cc:503] Master 127.0.103.126:37467 was elected leader, sending a full tablet report...
I20240430 17:04:11.727968 413 test_util.cc:268] Using random seed: 1541739595
I20240430 17:04:11.751274 3736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40836:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 17:04:11.753517 3736 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 17:04:11.805526 3903 tablet_service.cc:1467] Processing CreateTablet for tablet 6fcb848e4c1b4360a8cb22f4418a4afd (DEFAULT_TABLE table=test-workload [id=e2cc5c1bd4684a82b5abfba3f14aaa5d]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240430 17:04:11.806309 3902 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8fe0147cfb4fbf897f6644d81746c0 (DEFAULT_TABLE table=test-workload [id=e2cc5c1bd4684a82b5abfba3f14aaa5d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240430 17:04:11.806574 3901 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbf9f062d8d4528889226fa1567dfd6 (DEFAULT_TABLE table=test-workload [id=e2cc5c1bd4684a82b5abfba3f14aaa5d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240430 17:04:11.807001 3903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fcb848e4c1b4360a8cb22f4418a4afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:04:11.807771 3902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8fe0147cfb4fbf897f6644d81746c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:04:11.808802 3828 tablet_service.cc:1467] Processing CreateTablet for tablet 6fcb848e4c1b4360a8cb22f4418a4afd (DEFAULT_TABLE table=test-workload [id=e2cc5c1bd4684a82b5abfba3f14aaa5d]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240430 17:04:11.810076 3828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fcb848e4c1b4360a8cb22f4418a4afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:04:11.810798 3826 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbf9f062d8d4528889226fa1567dfd6 (DEFAULT_TABLE table=test-workload [id=e2cc5c1bd4684a82b5abfba3f14aaa5d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240430 17:04:11.812021 3826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbf9f062d8d4528889226fa1567dfd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:04:11.816032 3827 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8fe0147cfb4fbf897f6644d81746c0 (DEFAULT_TABLE table=test-workload [id=e2cc5c1bd4684a82b5abfba3f14aaa5d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240430 17:04:11.817309 3827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8fe0147cfb4fbf897f6644d81746c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:04:11.821655 3901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbf9f062d8d4528889226fa1567dfd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:04:11.825366 3977 tablet_service.cc:1467] Processing CreateTablet for tablet 6fcb848e4c1b4360a8cb22f4418a4afd (DEFAULT_TABLE table=test-workload [id=e2cc5c1bd4684a82b5abfba3f14aaa5d]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240430 17:04:11.826663 3977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fcb848e4c1b4360a8cb22f4418a4afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:04:11.832828 3976 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8fe0147cfb4fbf897f6644d81746c0 (DEFAULT_TABLE table=test-workload [id=e2cc5c1bd4684a82b5abfba3f14aaa5d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240430 17:04:11.834164 3976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8fe0147cfb4fbf897f6644d81746c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:04:11.836334 3975 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbf9f062d8d4528889226fa1567dfd6 (DEFAULT_TABLE table=test-workload [id=e2cc5c1bd4684a82b5abfba3f14aaa5d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240430 17:04:11.838160 4032 tablet_bootstrap.cc:492] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22: Bootstrap starting.
I20240430 17:04:11.842643 3975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbf9f062d8d4528889226fa1567dfd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:04:11.844491 4032 tablet_bootstrap.cc:654] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22: Neither blocks nor log segments found. Creating new log.
I20240430 17:04:11.868324 4033 tablet_bootstrap.cc:492] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f: Bootstrap starting.
I20240430 17:04:11.874630 4035 tablet_bootstrap.cc:492] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9: Bootstrap starting.
I20240430 17:04:11.875982 4033 tablet_bootstrap.cc:654] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f: Neither blocks nor log segments found. Creating new log.
I20240430 17:04:11.876493 4032 tablet_bootstrap.cc:492] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22: No bootstrap required, opened a new log
I20240430 17:04:11.876968 4032 ts_tablet_manager.cc:1367] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.024s
I20240430 17:04:11.879798 4032 raft_consensus.cc:348] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20240430 17:04:11.880538 4032 raft_consensus.cc:374] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:04:11.880712 4035 tablet_bootstrap.cc:654] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9: Neither blocks nor log segments found. Creating new log.
I20240430 17:04:11.880864 4032 raft_consensus.cc:729] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1488849a12e4e3dae956cd218f4ed22, State: Initialized, Role: FOLLOWER
I20240430 17:04:11.881623 4032 consensus_queue.cc:260] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20240430 17:04:11.886937 4033 tablet_bootstrap.cc:492] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f: No bootstrap required, opened a new log
I20240430 17:04:11.887375 4032 ts_tablet_manager.cc:1398] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20240430 17:04:11.887459 4033 ts_tablet_manager.cc:1367] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.007s
I20240430 17:04:11.888774 4032 tablet_bootstrap.cc:492] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22: Bootstrap starting.
I20240430 17:04:11.890530 4033 raft_consensus.cc:348] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20240430 17:04:11.891395 4033 raft_consensus.cc:374] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:04:11.891719 4033 raft_consensus.cc:729] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8e00d436a14372a8280644f779828f, State: Initialized, Role: FOLLOWER
I20240430 17:04:11.892544 4033 consensus_queue.cc:260] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20240430 17:04:11.896687 4032 tablet_bootstrap.cc:654] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22: Neither blocks nor log segments found. Creating new log.
I20240430 17:04:11.902992 4035 tablet_bootstrap.cc:492] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9: No bootstrap required, opened a new log
I20240430 17:04:11.902992 4032 tablet_bootstrap.cc:492] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22: No bootstrap required, opened a new log
I20240430 17:04:11.903678 4032 ts_tablet_manager.cc:1367] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20240430 17:04:11.903678 4035 ts_tablet_manager.cc:1367] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.012s
I20240430 17:04:11.903270 4033 ts_tablet_manager.cc:1398] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20240430 17:04:11.905619 4033 tablet_bootstrap.cc:492] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f: Bootstrap starting.
I20240430 17:04:11.906322 4035 raft_consensus.cc:348] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20240430 17:04:11.906824 4035 raft_consensus.cc:374] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:04:11.907014 4035 raft_consensus.cc:729] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b1d8e536e141a89a54b559d8a4a5f9, State: Initialized, Role: FOLLOWER
I20240430 17:04:11.907588 4035 consensus_queue.cc:260] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20240430 17:04:11.911185 4033 tablet_bootstrap.cc:654] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f: Neither blocks nor log segments found. Creating new log.
I20240430 17:04:11.911361 4032 raft_consensus.cc:348] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:11.912109 4032 raft_consensus.cc:374] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:04:11.912381 4032 raft_consensus.cc:729] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1488849a12e4e3dae956cd218f4ed22, State: Initialized, Role: FOLLOWER
I20240430 17:04:11.913055 4032 consensus_queue.cc:260] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:11.915225 4032 ts_tablet_manager.cc:1398] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20240430 17:04:11.916251 4032 tablet_bootstrap.cc:492] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22: Bootstrap starting.
I20240430 17:04:11.916309 4035 ts_tablet_manager.cc:1398] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9: Time spent starting tablet: real 0.012s user 0.001s sys 0.004s
I20240430 17:04:11.917305 4035 tablet_bootstrap.cc:492] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9: Bootstrap starting.
I20240430 17:04:11.917519 4033 tablet_bootstrap.cc:492] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f: No bootstrap required, opened a new log
I20240430 17:04:11.917989 4033 ts_tablet_manager.cc:1367] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240430 17:04:11.920562 4033 raft_consensus.cc:348] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:11.921265 4033 raft_consensus.cc:374] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:04:11.921540 4033 raft_consensus.cc:729] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8e00d436a14372a8280644f779828f, State: Initialized, Role: FOLLOWER
I20240430 17:04:11.922413 4032 tablet_bootstrap.cc:654] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22: Neither blocks nor log segments found. Creating new log.
I20240430 17:04:11.922698 4035 tablet_bootstrap.cc:654] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9: Neither blocks nor log segments found. Creating new log.
I20240430 17:04:11.922276 4033 consensus_queue.cc:260] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:11.924459 4033 ts_tablet_manager.cc:1398] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240430 17:04:11.925357 4033 tablet_bootstrap.cc:492] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f: Bootstrap starting.
I20240430 17:04:11.931208 4033 tablet_bootstrap.cc:654] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f: Neither blocks nor log segments found. Creating new log.
I20240430 17:04:11.958218 4032 tablet_bootstrap.cc:492] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22: No bootstrap required, opened a new log
I20240430 17:04:11.958732 4032 ts_tablet_manager.cc:1367] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22: Time spent bootstrapping tablet: real 0.043s user 0.013s sys 0.027s
I20240430 17:04:11.959424 4035 tablet_bootstrap.cc:492] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9: No bootstrap required, opened a new log
I20240430 17:04:11.959875 4033 tablet_bootstrap.cc:492] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f: No bootstrap required, opened a new log
I20240430 17:04:11.959908 4035 ts_tablet_manager.cc:1367] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9: Time spent bootstrapping tablet: real 0.043s user 0.017s sys 0.020s
I20240430 17:04:11.960382 4033 ts_tablet_manager.cc:1367] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.017s
I20240430 17:04:11.961366 4032 raft_consensus.cc:348] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:11.962133 4032 raft_consensus.cc:374] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:04:11.962409 4032 raft_consensus.cc:729] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1488849a12e4e3dae956cd218f4ed22, State: Initialized, Role: FOLLOWER
I20240430 17:04:11.962389 4033 raft_consensus.cc:348] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:11.963284 4033 raft_consensus.cc:374] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:04:11.963665 4033 raft_consensus.cc:729] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8e00d436a14372a8280644f779828f, State: Initialized, Role: FOLLOWER
I20240430 17:04:11.963337 4032 consensus_queue.cc:260] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:11.964147 4035 raft_consensus.cc:348] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:11.964239 4033 consensus_queue.cc:260] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:11.964941 4035 raft_consensus.cc:374] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:04:11.965278 4035 raft_consensus.cc:729] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b1d8e536e141a89a54b559d8a4a5f9, State: Initialized, Role: FOLLOWER
I20240430 17:04:11.965641 4032 ts_tablet_manager.cc:1398] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240430 17:04:11.966013 4035 consensus_queue.cc:260] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:11.966463 4033 ts_tablet_manager.cc:1398] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20240430 17:04:11.967967 4035 ts_tablet_manager.cc:1398] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20240430 17:04:11.968812 4035 tablet_bootstrap.cc:492] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9: Bootstrap starting.
I20240430 17:04:11.973405 4035 tablet_bootstrap.cc:654] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9: Neither blocks nor log segments found. Creating new log.
I20240430 17:04:11.981371 4035 tablet_bootstrap.cc:492] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9: No bootstrap required, opened a new log
I20240430 17:04:11.981741 4035 ts_tablet_manager.cc:1367] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20240430 17:04:11.983590 4035 raft_consensus.cc:348] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:11.984105 4035 raft_consensus.cc:374] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 17:04:11.984345 4035 raft_consensus.cc:729] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b1d8e536e141a89a54b559d8a4a5f9, State: Initialized, Role: FOLLOWER
I20240430 17:04:11.984862 4035 consensus_queue.cc:260] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:11.986650 4035 ts_tablet_manager.cc:1398] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240430 17:04:12.008301 4041 raft_consensus.cc:482] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:04:12.008715 4041 raft_consensus.cc:504] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:12.010660 4041 leader_election.cc:288] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761), 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435)
I20240430 17:04:12.017112 4037 raft_consensus.cc:482] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:04:12.017601 4037 raft_consensus.cc:504] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:12.020112 4037 leader_election.cc:288] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435), 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67:43323)
I20240430 17:04:12.024340 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8fe0147cfb4fbf897f6644d81746c0" candidate_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1488849a12e4e3dae956cd218f4ed22" is_pre_election: true
I20240430 17:04:12.025239 3838 raft_consensus.cc:2454] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b1d8e536e141a89a54b559d8a4a5f9 in term 0.
I20240430 17:04:12.026780 3952 leader_election.cc:302] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b1d8e536e141a89a54b559d8a4a5f9, f1488849a12e4e3dae956cd218f4ed22; no voters:
I20240430 17:04:12.027768 4041 raft_consensus.cc:2781] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:04:12.028117 4041 raft_consensus.cc:482] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:04:12.028439 4041 raft_consensus.cc:3037] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:04:12.034859 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8fe0147cfb4fbf897f6644d81746c0" candidate_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8e00d436a14372a8280644f779828f" is_pre_election: true
I20240430 17:04:12.035647 3913 raft_consensus.cc:2454] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b1d8e536e141a89a54b559d8a4a5f9 in term 0.
I20240430 17:04:12.035955 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbf9f062d8d4528889226fa1567dfd6" candidate_uuid: "f1488849a12e4e3dae956cd218f4ed22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8e00d436a14372a8280644f779828f" is_pre_election: true
I20240430 17:04:12.036628 3912 raft_consensus.cc:2454] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1488849a12e4e3dae956cd218f4ed22 in term 0.
I20240430 17:04:12.036602 4041 raft_consensus.cc:504] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:12.037838 3804 leader_election.cc:302] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8e00d436a14372a8280644f779828f, f1488849a12e4e3dae956cd218f4ed22; no voters:
I20240430 17:04:12.038545 4037 raft_consensus.cc:2781] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:04:12.038905 4037 raft_consensus.cc:482] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:04:12.039263 4037 raft_consensus.cc:3037] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:04:12.039497 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8fe0147cfb4fbf897f6644d81746c0" candidate_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1488849a12e4e3dae956cd218f4ed22"
I20240430 17:04:12.040059 3838 raft_consensus.cc:3037] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:04:12.042021 4041 leader_election.cc:288] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 1 election: Requested vote from peers f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761), 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435)
I20240430 17:04:12.042913 3987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbf9f062d8d4528889226fa1567dfd6" candidate_uuid: "f1488849a12e4e3dae956cd218f4ed22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" is_pre_election: true
I20240430 17:04:12.042949 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8fe0147cfb4fbf897f6644d81746c0" candidate_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8e00d436a14372a8280644f779828f"
I20240430 17:04:12.043713 3912 raft_consensus.cc:3037] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:04:12.043665 3987 raft_consensus.cc:2454] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1488849a12e4e3dae956cd218f4ed22 in term 0.
I20240430 17:04:12.046314 4037 raft_consensus.cc:504] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:12.047188 3838 raft_consensus.cc:2454] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b1d8e536e141a89a54b559d8a4a5f9 in term 1.
I20240430 17:04:12.048264 4037 leader_election.cc:288] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [CANDIDATE]: Term 1 election: Requested vote from peers 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435), 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67:43323)
I20240430 17:04:12.048429 3952 leader_election.cc:302] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b1d8e536e141a89a54b559d8a4a5f9, f1488849a12e4e3dae956cd218f4ed22; no voters:
I20240430 17:04:12.049264 4041 raft_consensus.cc:2781] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:04:12.049412 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbf9f062d8d4528889226fa1567dfd6" candidate_uuid: "f1488849a12e4e3dae956cd218f4ed22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8e00d436a14372a8280644f779828f"
I20240430 17:04:12.050060 3912 raft_consensus.cc:2454] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b1d8e536e141a89a54b559d8a4a5f9 in term 1.
I20240430 17:04:12.050065 3913 raft_consensus.cc:3037] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:04:12.050107 3987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbf9f062d8d4528889226fa1567dfd6" candidate_uuid: "f1488849a12e4e3dae956cd218f4ed22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b1d8e536e141a89a54b559d8a4a5f9"
I20240430 17:04:12.051141 3987 raft_consensus.cc:3037] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:04:12.052214 4041 raft_consensus.cc:686] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 LEADER]: Becoming Leader. State: Replica: 64b1d8e536e141a89a54b559d8a4a5f9, State: Running, Role: LEADER
I20240430 17:04:12.053676 4041 consensus_queue.cc:237] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:12.055519 3913 raft_consensus.cc:2454] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1488849a12e4e3dae956cd218f4ed22 in term 1.
I20240430 17:04:12.056380 3804 leader_election.cc:302] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8e00d436a14372a8280644f779828f, f1488849a12e4e3dae956cd218f4ed22; no voters:
I20240430 17:04:12.056995 4037 raft_consensus.cc:2781] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:04:12.058156 4037 raft_consensus.cc:686] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 1 LEADER]: Becoming Leader. State: Replica: f1488849a12e4e3dae956cd218f4ed22, State: Running, Role: LEADER
I20240430 17:04:12.058748 3987 raft_consensus.cc:2454] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1488849a12e4e3dae956cd218f4ed22 in term 1.
I20240430 17:04:12.058931 4037 consensus_queue.cc:237] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:12.065295 3735 catalog_manager.cc:5484] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } health_report { overall_health: HEALTHY } } }
I20240430 17:04:12.067432 3736 catalog_manager.cc:5484] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "f1488849a12e4e3dae956cd218f4ed22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } health_report { overall_health: UNKNOWN } } }
I20240430 17:04:12.126698 4037 raft_consensus.cc:482] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 17:04:12.127408 4037 raft_consensus.cc:504] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20240430 17:04:12.129411 4037 leader_election.cc:288] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435), 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67:43323)
I20240430 17:04:12.130147 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcb848e4c1b4360a8cb22f4418a4afd" candidate_uuid: "f1488849a12e4e3dae956cd218f4ed22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8e00d436a14372a8280644f779828f" is_pre_election: true
I20240430 17:04:12.130373 3987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcb848e4c1b4360a8cb22f4418a4afd" candidate_uuid: "f1488849a12e4e3dae956cd218f4ed22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" is_pre_election: true
I20240430 17:04:12.130751 3913 raft_consensus.cc:2454] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1488849a12e4e3dae956cd218f4ed22 in term 0.
I20240430 17:04:12.130980 3987 raft_consensus.cc:2454] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1488849a12e4e3dae956cd218f4ed22 in term 0.
I20240430 17:04:12.131750 3804 leader_election.cc:302] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8e00d436a14372a8280644f779828f, f1488849a12e4e3dae956cd218f4ed22; no voters:
I20240430 17:04:12.132448 4037 raft_consensus.cc:2781] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 17:04:12.132782 4037 raft_consensus.cc:482] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 17:04:12.133033 4037 raft_consensus.cc:3037] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:04:12.137404 4037 raft_consensus.cc:504] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20240430 17:04:12.138833 4037 leader_election.cc:288] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [CANDIDATE]: Term 1 election: Requested vote from peers 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435), 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67:43323)
I20240430 17:04:12.139530 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcb848e4c1b4360a8cb22f4418a4afd" candidate_uuid: "f1488849a12e4e3dae956cd218f4ed22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8e00d436a14372a8280644f779828f"
I20240430 17:04:12.139740 3987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcb848e4c1b4360a8cb22f4418a4afd" candidate_uuid: "f1488849a12e4e3dae956cd218f4ed22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b1d8e536e141a89a54b559d8a4a5f9"
I20240430 17:04:12.140028 3913 raft_consensus.cc:3037] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:04:12.140223 3987 raft_consensus.cc:3037] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 0 FOLLOWER]: Advancing to term 1
I20240430 17:04:12.144474 3913 raft_consensus.cc:2454] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1488849a12e4e3dae956cd218f4ed22 in term 1.
I20240430 17:04:12.144623 3987 raft_consensus.cc:2454] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1488849a12e4e3dae956cd218f4ed22 in term 1.
I20240430 17:04:12.145341 3804 leader_election.cc:302] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8e00d436a14372a8280644f779828f, f1488849a12e4e3dae956cd218f4ed22; no voters:
I20240430 17:04:12.146041 4037 raft_consensus.cc:2781] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 17:04:12.146443 4037 raft_consensus.cc:686] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 1 LEADER]: Becoming Leader. State: Replica: f1488849a12e4e3dae956cd218f4ed22, State: Running, Role: LEADER
I20240430 17:04:12.147034 4037 consensus_queue.cc:237] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20240430 17:04:12.153760 3736 catalog_manager.cc:5484] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "f1488849a12e4e3dae956cd218f4ed22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } health_report { overall_health: HEALTHY } } }
I20240430 17:04:12.180121 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 17:04:12.186200 4057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:04:12.188699 4058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 17:04:12.189395 4060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 17:04:12.190308 413 server_base.cc:1034] running on GCE node
I20240430 17:04:12.191432 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240430 17:04:12.191617 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240430 17:04:12.191745 413 hybrid_clock.cc:648] HybridClock initialized: now 1714496652191735 us; error 0 us; skew 500 ppm
I20240430 17:04:12.192184 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 17:04:12.194427 413 webserver.cc:460] Webserver started at http://127.0.103.68:33413/ using document root <none> and password file <none>
I20240430 17:04:12.194875 413 fs_manager.cc:359] Metadata directory not provided
I20240430 17:04:12.195055 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 17:04:12.195366 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 17:04:12.196435 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-3-root/instance:
uuid: "3fd7cfcf66b640158a1b2f30112ca89b"
format_stamp: "Formatted at 2024-04-30 17:04:12 on dist-test-slave-x21r"
I20240430 17:04:12.200570 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240430 17:04:12.203508 4065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:12.204176 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240430 17:04:12.204447 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-3-root
uuid: "3fd7cfcf66b640158a1b2f30112ca89b"
format_stamp: "Formatted at 2024-04-30 17:04:12 on dist-test-slave-x21r"
I20240430 17:04:12.204704 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 17:04:12.214043 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 17:04:12.215093 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 17:04:12.216461 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 17:04:12.218565 413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 17:04:12.218766 413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:12.218968 413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 17:04:12.219156 413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 17:04:12.259403 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:44271
I20240430 17:04:12.259495 4127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.103.68:44271 every 8 connection(s)
I20240430 17:04:12.272560 4128 heartbeater.cc:347] Connected to a master server at 127.0.103.126:37467
I20240430 17:04:12.272894 4128 heartbeater.cc:464] Registering TS with master...
I20240430 17:04:12.273561 4128 heartbeater.cc:511] Master 127.0.103.126:37467 requested a full tablet report, sending...
I20240430 17:04:12.275203 3736 ts_manager.cc:194] Registered new tserver with Master: 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68:44271)
I20240430 17:04:12.276473 3736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40852
I20240430 17:04:12.291457 4037 consensus_queue.cc:1035] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 17:04:12.297518 4054 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240430 17:04:12.297798 4054 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240430 17:04:12.339030 4037 consensus_queue.cc:1035] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:04:12.341938 4041 consensus_queue.cc:1035] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 17:04:12.347652 4041 consensus_queue.cc:1035] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:04:12.356381 3789 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:04:12.386349 4037 consensus_queue.cc:1035] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:04:12.394188 3838 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6fcb848e4c1b4360a8cb22f4418a4afd"
dest_uuid: "f1488849a12e4e3dae956cd218f4ed22"
mode: GRACEFUL
new_leader_uuid: "5e8e00d436a14372a8280644f779828f"
from {username='slave'} at 127.0.0.1:47516
I20240430 17:04:12.395759 3837 consensus_queue.cc:237] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } }
I20240430 17:04:12.398849 3838 raft_consensus.cc:595] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 1 LEADER]: Received request to transfer leadership to TS 5e8e00d436a14372a8280644f779828f
I20240430 17:04:12.403322 3987 raft_consensus.cc:1261] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Refusing update from remote peer f1488849a12e4e3dae956cd218f4ed22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:04:12.405742 4037 consensus_queue.cc:1035] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 17:04:12.406795 3789 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240430 17:04:12.407444 3789 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:04:12.407606 3913 raft_consensus.cc:1261] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Refusing update from remote peer f1488849a12e4e3dae956cd218f4ed22: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:04:12.409047 4037 consensus_queue.cc:1035] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
W20240430 17:04:12.418649 3804 consensus_peers.cc:485] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 -> Peer 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68:44271): Couldn't send request to peer 3fd7cfcf66b640158a1b2f30112ca89b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fcb848e4c1b4360a8cb22f4418a4afd. This is attempt 1: this message will repeat every 5th retry.
I20240430 17:04:12.419178 3987 raft_consensus.cc:2932] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } } }
I20240430 17:04:12.421039 3913 raft_consensus.cc:2932] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } } }
I20240430 17:04:12.428530 3736 catalog_manager.cc:5484] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "f1488849a12e4e3dae956cd218f4ed22" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } } }
I20240430 17:04:12.665246 4143 raft_consensus.cc:2932] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } } }
I20240430 17:04:12.683614 3987 consensus_queue.cc:237] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } }
I20240430 17:04:12.691818 3838 raft_consensus.cc:1261] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Refusing update from remote peer 64b1d8e536e141a89a54b559d8a4a5f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:04:12.694671 3913 raft_consensus.cc:1261] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Refusing update from remote peer 64b1d8e536e141a89a54b559d8a4a5f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 17:04:12.695896 4153 ts_tablet_manager.cc:902] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b: Initiating tablet copy from peer f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761)
I20240430 17:04:12.698976 4052 consensus_queue.cc:1035] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 17:04:12.707077 4052 consensus_queue.cc:1035] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240430 17:04:12.708274 3953 consensus_peers.cc:485] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 -> Peer 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68:44271): Couldn't send request to peer 3fd7cfcf66b640158a1b2f30112ca89b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a8fe0147cfb4fbf897f6644d81746c0. This is attempt 1: this message will repeat every 5th retry.
I20240430 17:04:12.707617 3838 raft_consensus.cc:2932] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } } }
I20240430 17:04:12.712513 3913 raft_consensus.cc:2932] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } } }
I20240430 17:04:12.715369 4153 tablet_copy_client.cc:308] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:38761
I20240430 17:04:12.721868 3735 catalog_manager.cc:5484] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 reported cstate change: config changed from index -1 to 2, NON_VOTER 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } } }
I20240430 17:04:12.728209 3848 tablet_copy_service.cc:140] P f1488849a12e4e3dae956cd218f4ed22: Received BeginTabletCopySession request for tablet 6fcb848e4c1b4360a8cb22f4418a4afd from peer 3fd7cfcf66b640158a1b2f30112ca89b ({username='slave'} at 127.0.0.1:47528)
I20240430 17:04:12.728641 3848 tablet_copy_service.cc:161] P f1488849a12e4e3dae956cd218f4ed22: Beginning new tablet copy session on tablet 6fcb848e4c1b4360a8cb22f4418a4afd from peer 3fd7cfcf66b640158a1b2f30112ca89b at {username='slave'} at 127.0.0.1:47528: session id = 3fd7cfcf66b640158a1b2f30112ca89b-6fcb848e4c1b4360a8cb22f4418a4afd
I20240430 17:04:12.733829 3848 tablet_copy_source_session.cc:205] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22: Tablet Copy: opened 0 blocks and 1 log segments
I20240430 17:04:12.736271 4153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fcb848e4c1b4360a8cb22f4418a4afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:04:12.746057 4153 tablet_copy_client.cc:783] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b: tablet copy: Starting download of 0 data blocks...
I20240430 17:04:12.746560 4153 tablet_copy_client.cc:647] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b: tablet copy: Starting download of 1 WAL segments...
I20240430 17:04:12.749500 4153 tablet_copy_client.cc:523] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240430 17:04:12.754981 4153 tablet_bootstrap.cc:492] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b: Bootstrap starting.
I20240430 17:04:12.768708 4153 tablet_bootstrap.cc:492] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240430 17:04:12.769317 4153 tablet_bootstrap.cc:492] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b: Bootstrap complete.
I20240430 17:04:12.769774 4153 ts_tablet_manager.cc:1367] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.004s
I20240430 17:04:12.771633 4153 raft_consensus.cc:348] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } }
I20240430 17:04:12.772249 4153 raft_consensus.cc:729] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fd7cfcf66b640158a1b2f30112ca89b, State: Initialized, Role: LEARNER
I20240430 17:04:12.772734 4153 consensus_queue.cc:260] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } }
I20240430 17:04:12.774411 4128 heartbeater.cc:503] Master 127.0.103.126:37467 was elected leader, sending a full tablet report...
I20240430 17:04:12.774778 4153 ts_tablet_manager.cc:1398] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240430 17:04:12.776566 3848 tablet_copy_service.cc:342] P f1488849a12e4e3dae956cd218f4ed22: Request end of tablet copy session 3fd7cfcf66b640158a1b2f30112ca89b-6fcb848e4c1b4360a8cb22f4418a4afd received from {username='slave'} at 127.0.0.1:47528
I20240430 17:04:12.776950 3848 tablet_copy_service.cc:433] P f1488849a12e4e3dae956cd218f4ed22: ending tablet copy session 3fd7cfcf66b640158a1b2f30112ca89b-6fcb848e4c1b4360a8cb22f4418a4afd on tablet 6fcb848e4c1b4360a8cb22f4418a4afd with peer 3fd7cfcf66b640158a1b2f30112ca89b
I20240430 17:04:12.859758 4147 raft_consensus.cc:2932] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } } }
I20240430 17:04:12.959244 4103 raft_consensus.cc:1203] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240430 17:04:12.994047 4143 raft_consensus.cc:979] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22: : Instructing follower 5e8e00d436a14372a8280644f779828f to start an election
I20240430 17:04:12.994679 4148 raft_consensus.cc:1067] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 1 LEADER]: Signalling peer 5e8e00d436a14372a8280644f779828f to start an election
I20240430 17:04:12.996171 3913 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6fcb848e4c1b4360a8cb22f4418a4afd"
dest_uuid: "5e8e00d436a14372a8280644f779828f"
from {username='slave'} at 127.0.0.1:42016
I20240430 17:04:12.996675 3913 raft_consensus.cc:482] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240430 17:04:12.997028 3913 raft_consensus.cc:3037] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Advancing to term 2
I20240430 17:04:13.002372 3913 raft_consensus.cc:504] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } }
I20240430 17:04:13.006376 3913 leader_election.cc:288] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [CANDIDATE]: Term 2 election: Requested vote from peers 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67:43323), f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761)
I20240430 17:04:13.015460 3986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcb848e4c1b4360a8cb22f4418a4afd" candidate_uuid: "5e8e00d436a14372a8280644f779828f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "64b1d8e536e141a89a54b559d8a4a5f9"
I20240430 17:04:13.016088 3986 raft_consensus.cc:3037] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Advancing to term 2
I20240430 17:04:13.016503 3838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcb848e4c1b4360a8cb22f4418a4afd" candidate_uuid: "5e8e00d436a14372a8280644f779828f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "f1488849a12e4e3dae956cd218f4ed22"
I20240430 17:04:13.017045 3838 raft_consensus.cc:3032] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 17:04:13.017357 3838 raft_consensus.cc:729] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f1488849a12e4e3dae956cd218f4ed22, State: Running, Role: LEADER
I20240430 17:04:13.018113 3838 consensus_queue.cc:260] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } }
I20240430 17:04:13.019655 3838 raft_consensus.cc:3037] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Advancing to term 2
I20240430 17:04:13.020792 3913 raft_consensus.cc:1226] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 2 FOLLOWER]: Rejecting Update request from peer f1488849a12e4e3dae956cd218f4ed22 for earlier term 1. Current term is 2. Ops: []
I20240430 17:04:13.022282 3986 raft_consensus.cc:2454] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8e00d436a14372a8280644f779828f in term 2.
I20240430 17:04:13.023265 3876 leader_election.cc:302] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8e00d436a14372a8280644f779828f, 64b1d8e536e141a89a54b559d8a4a5f9; no voters:
I20240430 17:04:13.024209 4039 raft_consensus.cc:2781] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 2 FOLLOWER]: Leader election won for term 2
I20240430 17:04:13.025305 4039 raft_consensus.cc:686] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 2 LEADER]: Becoming Leader. State: Replica: 5e8e00d436a14372a8280644f779828f, State: Running, Role: LEADER
I20240430 17:04:13.025422 3838 raft_consensus.cc:2454] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8e00d436a14372a8280644f779828f in term 2.
I20240430 17:04:13.026176 4039 consensus_queue.cc:237] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } }
I20240430 17:04:13.033681 3733 catalog_manager.cc:5484] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f reported cstate change: term changed from 1 to 2, leader changed from f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65) to 5e8e00d436a14372a8280644f779828f (127.0.103.66). New cstate: current_term: 2 leader_uuid: "5e8e00d436a14372a8280644f779828f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240430 17:04:13.064811 4153 ts_tablet_manager.cc:902] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b: Initiating tablet copy from peer 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67:43323)
I20240430 17:04:13.068323 4153 tablet_copy_client.cc:308] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:43323
I20240430 17:04:13.076925 3997 tablet_copy_service.cc:140] P 64b1d8e536e141a89a54b559d8a4a5f9: Received BeginTabletCopySession request for tablet 4a8fe0147cfb4fbf897f6644d81746c0 from peer 3fd7cfcf66b640158a1b2f30112ca89b ({username='slave'} at 127.0.0.1:60614)
I20240430 17:04:13.077431 3997 tablet_copy_service.cc:161] P 64b1d8e536e141a89a54b559d8a4a5f9: Beginning new tablet copy session on tablet 4a8fe0147cfb4fbf897f6644d81746c0 from peer 3fd7cfcf66b640158a1b2f30112ca89b at {username='slave'} at 127.0.0.1:60614: session id = 3fd7cfcf66b640158a1b2f30112ca89b-4a8fe0147cfb4fbf897f6644d81746c0
I20240430 17:04:13.083681 3997 tablet_copy_source_session.cc:205] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240430 17:04:13.086369 4153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8fe0147cfb4fbf897f6644d81746c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 17:04:13.098356 4153 tablet_copy_client.cc:783] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b: tablet copy: Starting download of 0 data blocks...
I20240430 17:04:13.098896 4153 tablet_copy_client.cc:647] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b: tablet copy: Starting download of 1 WAL segments...
I20240430 17:04:13.102053 4153 tablet_copy_client.cc:523] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240430 17:04:13.108340 4153 tablet_bootstrap.cc:492] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b: Bootstrap starting.
I20240430 17:04:13.123302 4153 tablet_bootstrap.cc:492] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240430 17:04:13.123934 4153 tablet_bootstrap.cc:492] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b: Bootstrap complete.
I20240430 17:04:13.124430 4153 ts_tablet_manager.cc:1367] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20240430 17:04:13.126432 4153 raft_consensus.cc:348] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } }
I20240430 17:04:13.127300 4153 raft_consensus.cc:729] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fd7cfcf66b640158a1b2f30112ca89b, State: Initialized, Role: LEARNER
I20240430 17:04:13.127797 4153 consensus_queue.cc:260] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true } }
I20240430 17:04:13.129631 4153 ts_tablet_manager.cc:1398] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240430 17:04:13.131191 3997 tablet_copy_service.cc:342] P 64b1d8e536e141a89a54b559d8a4a5f9: Request end of tablet copy session 3fd7cfcf66b640158a1b2f30112ca89b-4a8fe0147cfb4fbf897f6644d81746c0 received from {username='slave'} at 127.0.0.1:60614
I20240430 17:04:13.131551 3997 tablet_copy_service.cc:433] P 64b1d8e536e141a89a54b559d8a4a5f9: ending tablet copy session 3fd7cfcf66b640158a1b2f30112ca89b-4a8fe0147cfb4fbf897f6644d81746c0 on tablet 4a8fe0147cfb4fbf897f6644d81746c0 with peer 3fd7cfcf66b640158a1b2f30112ca89b
I20240430 17:04:13.260185 3838 raft_consensus.cc:1261] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 2 FOLLOWER]: Refusing update from remote peer 5e8e00d436a14372a8280644f779828f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240430 17:04:13.261397 4039 consensus_queue.cc:1035] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 17:04:13.269662 3986 raft_consensus.cc:1261] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 2 FOLLOWER]: Refusing update from remote peer 5e8e00d436a14372a8280644f779828f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240430 17:04:13.270977 4162 consensus_queue.cc:1035] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 17:04:13.365542 4103 raft_consensus.cc:3037] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 LEARNER]: Advancing to term 2
I20240430 17:04:13.368162 4102 raft_consensus.cc:1203] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240430 17:04:13.372272 4103 raft_consensus.cc:1261] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 LEARNER]: Refusing update from remote peer 5e8e00d436a14372a8280644f779828f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240430 17:04:13.373440 4162 consensus_queue.cc:1035] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 17:04:13.923008 4039 raft_consensus.cc:1050] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f: attempting to promote NON_VOTER 3fd7cfcf66b640158a1b2f30112ca89b to VOTER
I20240430 17:04:13.925163 4039 consensus_queue.cc:237] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } }
I20240430 17:04:13.931001 3838 raft_consensus.cc:1261] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 2 FOLLOWER]: Refusing update from remote peer 5e8e00d436a14372a8280644f779828f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240430 17:04:13.931082 4103 raft_consensus.cc:1261] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 LEARNER]: Refusing update from remote peer 5e8e00d436a14372a8280644f779828f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240430 17:04:13.932581 3987 raft_consensus.cc:1261] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 2 FOLLOWER]: Refusing update from remote peer 5e8e00d436a14372a8280644f779828f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240430 17:04:13.932621 4039 consensus_queue.cc:1035] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240430 17:04:13.934018 4162 consensus_queue.cc:1035] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240430 17:04:13.934857 4176 consensus_queue.cc:1035] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240430 17:04:13.944319 4151 raft_consensus.cc:1050] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9: attempting to promote NON_VOTER 3fd7cfcf66b640158a1b2f30112ca89b to VOTER
I20240430 17:04:13.946662 4151 consensus_queue.cc:237] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } }
I20240430 17:04:13.949319 3986 raft_consensus.cc:2932] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:13.954203 4103 raft_consensus.cc:1261] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 LEARNER]: Refusing update from remote peer 64b1d8e536e141a89a54b559d8a4a5f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240430 17:04:13.955294 3912 raft_consensus.cc:1261] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Refusing update from remote peer 64b1d8e536e141a89a54b559d8a4a5f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240430 17:04:13.956460 3838 raft_consensus.cc:1261] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Refusing update from remote peer 64b1d8e536e141a89a54b559d8a4a5f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240430 17:04:13.956811 4147 consensus_queue.cc:1035] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 17:04:13.958530 4179 consensus_queue.cc:1035] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 17:04:13.959446 4147 consensus_queue.cc:1035] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 17:04:13.959261 4103 raft_consensus.cc:2932] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:13.968062 3733 catalog_manager.cc:5484] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 reported cstate change: config changed from index 2 to 4, 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "5e8e00d436a14372a8280644f779828f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:13.985452 3912 raft_consensus.cc:2932] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:13.985649 4102 raft_consensus.cc:2932] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.002954 3735 catalog_manager.cc:5484] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f reported cstate change: config changed from index 2 to 3, 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.109810 4147 raft_consensus.cc:2932] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.117587 3838 raft_consensus.cc:2932] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } attrs { replace: true } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.121654 4039 raft_consensus.cc:1013] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 2 LEADER]: attempt to promote peer 3fd7cfcf66b640158a1b2f30112ca89b: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240430 17:04:14.177278 3986 consensus_queue.cc:237] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } }
I20240430 17:04:14.182267 4102 raft_consensus.cc:1261] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 FOLLOWER]: Refusing update from remote peer 64b1d8e536e141a89a54b559d8a4a5f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240430 17:04:14.182351 3838 raft_consensus.cc:1261] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Refusing update from remote peer 64b1d8e536e141a89a54b559d8a4a5f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240430 17:04:14.183568 4151 consensus_queue.cc:1035] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240430 17:04:14.184399 4179 consensus_queue.cc:1035] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240430 17:04:14.191692 3838 raft_consensus.cc:2932] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8e00d436a14372a8280644f779828f (127.0.103.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.191879 4102 raft_consensus.cc:2932] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8e00d436a14372a8280644f779828f (127.0.103.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.202435 3733 catalog_manager.cc:5484] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b reported cstate change: config changed from index 3 to 4, VOTER 5e8e00d436a14372a8280644f779828f (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.211035 3838 raft_consensus.cc:2932] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.387023 4039 raft_consensus.cc:1013] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 2 LEADER]: attempt to promote peer 3fd7cfcf66b640158a1b2f30112ca89b: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240430 17:04:14.408381 3789 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:04:14.411804 3789 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:04:14.412171 3789 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:04:14.432485 3903 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a8fe0147cfb4fbf897f6644d81746c0 with delete_type TABLET_DATA_TOMBSTONED (TS 5e8e00d436a14372a8280644f779828f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41986
I20240430 17:04:14.434322 4187 tablet_replica.cc:330] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f: stopping tablet replica
I20240430 17:04:14.437141 4187 raft_consensus.cc:2229] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:04:14.437949 4187 raft_consensus.cc:2258] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:14.441057 4187 ts_tablet_manager.cc:1875] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240430 17:04:14.453481 4187 ts_tablet_manager.cc:1888] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240430 17:04:14.453773 4187 log.cc:1196] T 4a8fe0147cfb4fbf897f6644d81746c0 P 5e8e00d436a14372a8280644f779828f: Deleting WAL directory at /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-1-root/wals/4a8fe0147cfb4fbf897f6644d81746c0
I20240430 17:04:14.455195 3723 catalog_manager.cc:4830] TS 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435): tablet 4a8fe0147cfb4fbf897f6644d81746c0 (table test-workload [id=e2cc5c1bd4684a82b5abfba3f14aaa5d]) successfully deleted
I20240430 17:04:14.511546 4178 raft_consensus.cc:2932] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5e8e00d436a14372a8280644f779828f (127.0.103.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.518379 3720 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4a8fe0147cfb4fbf897f6644d81746c0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240430 17:04:14.544592 4172 raft_consensus.cc:2932] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3fd7cfcf66b640158a1b2f30112ca89b (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } attrs { replace: true } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.557963 3912 consensus_queue.cc:237] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } }
I20240430 17:04:14.561661 4102 raft_consensus.cc:1261] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Refusing update from remote peer 5e8e00d436a14372a8280644f779828f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240430 17:04:14.562050 3838 raft_consensus.cc:1261] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 2 FOLLOWER]: Refusing update from remote peer 5e8e00d436a14372a8280644f779828f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240430 17:04:14.562893 4039 consensus_queue.cc:1035] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240430 17:04:14.563627 4176 consensus_queue.cc:1035] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240430 17:04:14.570483 3838 raft_consensus.cc:2932] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.571794 4102 raft_consensus.cc:2932] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.580574 3734 catalog_manager.cc:5484] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 reported cstate change: config changed from index 4 to 5, VOTER 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "5e8e00d436a14372a8280644f779828f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:14.842869 3977 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fcb848e4c1b4360a8cb22f4418a4afd with delete_type TABLET_DATA_TOMBSTONED (TS 64b1d8e536e141a89a54b559d8a4a5f9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60576
I20240430 17:04:14.844396 4188 tablet_replica.cc:330] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9: stopping tablet replica
I20240430 17:04:14.845027 4188 raft_consensus.cc:2229] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 17:04:14.845700 4188 raft_consensus.cc:2258] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:14.848402 4188 ts_tablet_manager.cc:1875] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240430 17:04:14.862737 4188 ts_tablet_manager.cc:1888] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240430 17:04:14.863063 4188 log.cc:1196] T 6fcb848e4c1b4360a8cb22f4418a4afd P 64b1d8e536e141a89a54b559d8a4a5f9: Deleting WAL directory at /tmp/dist-test-taskIaYKCP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714496547126677-413-0/minicluster-data/ts-2-root/wals/6fcb848e4c1b4360a8cb22f4418a4afd
I20240430 17:04:14.864373 3720 catalog_manager.cc:4830] TS 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67:43323): tablet 6fcb848e4c1b4360a8cb22f4418a4afd (table test-workload [id=e2cc5c1bd4684a82b5abfba3f14aaa5d]) successfully deleted
I20240430 17:04:15.195338 4039 raft_consensus.cc:2932] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 64b1d8e536e141a89a54b559d8a4a5f9 (127.0.103.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } } }
I20240430 17:04:15.203908 3723 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6fcb848e4c1b4360a8cb22f4418a4afd with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240430 17:04:16.413020 3789 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:04:16.416271 3789 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:04:16.416739 3789 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:04:18.417831 3789 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:04:18.420698 3789 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:04:18.421089 3789 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:04:20.690943 3789 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:04:20.696063 3789 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:04:20.696682 3789 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:04:20.766428 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20240430 17:04:20.810238 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:04:20.810899 413 tablet_replica.cc:330] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22: stopping tablet replica
I20240430 17:04:20.811590 413 raft_consensus.cc:2229] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:04:20.812672 413 raft_consensus.cc:2258] T 4a8fe0147cfb4fbf897f6644d81746c0 P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:20.814864 413 tablet_replica.cc:330] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22: stopping tablet replica
I20240430 17:04:20.815395 413 raft_consensus.cc:2229] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:04:20.816304 413 raft_consensus.cc:2258] T 7dbf9f062d8d4528889226fa1567dfd6 P f1488849a12e4e3dae956cd218f4ed22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:20.818333 413 tablet_replica.cc:330] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22: stopping tablet replica
I20240430 17:04:20.818785 413 raft_consensus.cc:2229] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 17:04:20.819793 413 raft_consensus.cc:2258] T 6fcb848e4c1b4360a8cb22f4418a4afd P f1488849a12e4e3dae956cd218f4ed22 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:20.844584 413 tablet_server.cc:194] TabletServer@127.0.103.65:0 shutdown complete.
I20240430 17:04:20.860033 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20240430 17:04:20.882220 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:04:20.883008 413 tablet_replica.cc:330] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f: stopping tablet replica
I20240430 17:04:20.883634 413 raft_consensus.cc:2229] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:04:20.884096 413 raft_consensus.cc:2258] T 7dbf9f062d8d4528889226fa1567dfd6 P 5e8e00d436a14372a8280644f779828f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:20.886222 413 tablet_replica.cc:330] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f: stopping tablet replica
I20240430 17:04:20.886669 413 raft_consensus.cc:2229] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 2 LEADER]: Raft consensus shutting down.
I20240430 17:04:20.887552 413 raft_consensus.cc:2258] T 6fcb848e4c1b4360a8cb22f4418a4afd P 5e8e00d436a14372a8280644f779828f [term 2 FOLLOWER]: Raft consensus is shut down!
W20240430 17:04:20.936094 3952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38761: connect: Connection refused (error 111)
W20240430 17:04:20.939234 3952 consensus_peers.cc:485] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 -> Peer f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761): Couldn't send request to peer f1488849a12e4e3dae956cd218f4ed22. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38761: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 17:04:21.591976 4192 raft_consensus.cc:482] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1488849a12e4e3dae956cd218f4ed22)
I20240430 17:04:21.592454 4192 raft_consensus.cc:504] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:21.594251 4192 leader_election.cc:288] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761), 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435)
W20240430 17:04:21.597529 3952 leader_election.cc:334] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761): Network error: Client connection negotiation failed: client connection to 127.0.103.65:38761: connect: Connection refused (error 111)
W20240430 17:04:21.598376 3953 leader_election.cc:334] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435): Network error: Client connection negotiation failed: client connection to 127.0.103.66:36435: connect: Connection refused (error 111)
I20240430 17:04:21.598712 3953 leader_election.cc:302] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64b1d8e536e141a89a54b559d8a4a5f9; no voters: 5e8e00d436a14372a8280644f779828f, f1488849a12e4e3dae956cd218f4ed22
I20240430 17:04:21.599305 4192 raft_consensus.cc:2726] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 17:04:21.694145 4198 raft_consensus.cc:482] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8e00d436a14372a8280644f779828f)
I20240430 17:04:21.694659 4198 raft_consensus.cc:504] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } }
I20240430 17:04:21.696581 4198 leader_election.cc:288] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435), f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761)
W20240430 17:04:21.702090 4069 leader_election.cc:334] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435): Network error: Client connection negotiation failed: client connection to 127.0.103.66:36435: connect: Connection refused (error 111)
W20240430 17:04:21.702656 4068 leader_election.cc:334] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761): Network error: Client connection negotiation failed: client connection to 127.0.103.65:38761: connect: Connection refused (error 111)
I20240430 17:04:21.703001 4068 leader_election.cc:302] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fd7cfcf66b640158a1b2f30112ca89b; no voters: 5e8e00d436a14372a8280644f779828f, f1488849a12e4e3dae956cd218f4ed22
I20240430 17:04:21.703610 4198 raft_consensus.cc:2726] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240430 17:04:21.726950 3934 debug-util.cc:398] Leaking SignalData structure 0x7b080008a9e0 after lost signal to thread 416
W20240430 17:04:21.727810 3934 debug-util.cc:398] Leaking SignalData structure 0x7b0800152480 after lost signal to thread 3770
W20240430 17:04:21.729887 3934 debug-util.cc:398] Leaking SignalData structure 0x7b080011bc20 after lost signal to thread 4011
W20240430 17:04:21.730718 3934 debug-util.cc:398] Leaking SignalData structure 0x7b08001f68a0 after lost signal to thread 4127
W20240430 17:04:21.909524 413 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 3934)
W20240430 17:04:22.369796 3952 consensus_peers.cc:485] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 -> Peer f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761): Couldn't send request to peer f1488849a12e4e3dae956cd218f4ed22. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 17:04:22.638948 4202 raft_consensus.cc:482] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8e00d436a14372a8280644f779828f)
I20240430 17:04:22.639463 4202 raft_consensus.cc:504] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } }
I20240430 17:04:22.641069 4202 leader_election.cc:288] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435), f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761)
W20240430 17:04:22.651659 4069 leader_election.cc:334] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435): Network error: Client connection negotiation failed: client connection to 127.0.103.66:36435: connect: Connection refused (error 111)
W20240430 17:04:22.652177 4068 leader_election.cc:334] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761): Network error: Client connection negotiation failed: client connection to 127.0.103.65:38761: connect: Connection refused (error 111)
I20240430 17:04:22.652520 4068 leader_election.cc:302] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fd7cfcf66b640158a1b2f30112ca89b; no voters: 5e8e00d436a14372a8280644f779828f, f1488849a12e4e3dae956cd218f4ed22
I20240430 17:04:22.653098 4202 raft_consensus.cc:2726] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240430 17:04:22.697558 3789 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240430 17:04:22.700559 3789 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240430 17:04:22.701097 3789 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240430 17:04:22.813577 4192 raft_consensus.cc:482] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1488849a12e4e3dae956cd218f4ed22)
I20240430 17:04:22.814034 4192 raft_consensus.cc:504] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:22.815627 4192 leader_election.cc:288] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761), 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435)
W20240430 17:04:22.820031 3952 leader_election.cc:334] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761): Network error: Client connection negotiation failed: client connection to 127.0.103.65:38761: connect: Connection refused (error 111)
W20240430 17:04:22.820575 3953 leader_election.cc:334] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435): Network error: Client connection negotiation failed: client connection to 127.0.103.66:36435: connect: Connection refused (error 111)
I20240430 17:04:22.820923 3953 leader_election.cc:302] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64b1d8e536e141a89a54b559d8a4a5f9; no voters: 5e8e00d436a14372a8280644f779828f, f1488849a12e4e3dae956cd218f4ed22
I20240430 17:04:22.821470 4192 raft_consensus.cc:2726] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 17:04:22.909950 413 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 3934)
W20240430 17:04:23.767434 3952 consensus_peers.cc:485] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 -> Peer f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761): Couldn't send request to peer f1488849a12e4e3dae956cd218f4ed22. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240430 17:04:23.910456 413 thread.cc:536] Waited for 3000ms trying to join with diag-logger (tid 3934)
I20240430 17:04:24.072168 4207 raft_consensus.cc:482] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e8e00d436a14372a8280644f779828f)
I20240430 17:04:24.072613 4207 raft_consensus.cc:504] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "3fd7cfcf66b640158a1b2f30112ca89b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44271 } attrs { promote: false } }
I20240430 17:04:24.074301 4207 leader_election.cc:288] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435), f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761)
W20240430 17:04:24.079908 4069 leader_election.cc:334] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435): Network error: Client connection negotiation failed: client connection to 127.0.103.66:36435: connect: Connection refused (error 111)
W20240430 17:04:24.080535 4068 leader_election.cc:334] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761): Network error: Client connection negotiation failed: client connection to 127.0.103.65:38761: connect: Connection refused (error 111)
I20240430 17:04:24.081027 4068 leader_election.cc:302] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fd7cfcf66b640158a1b2f30112ca89b; no voters: 5e8e00d436a14372a8280644f779828f, f1488849a12e4e3dae956cd218f4ed22
I20240430 17:04:24.081784 4207 raft_consensus.cc:2726] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240430 17:04:24.472255 4192 raft_consensus.cc:482] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1488849a12e4e3dae956cd218f4ed22)
I20240430 17:04:24.472847 4192 raft_consensus.cc:504] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1488849a12e4e3dae956cd218f4ed22" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "5e8e00d436a14372a8280644f779828f" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36435 } } peers { permanent_uuid: "64b1d8e536e141a89a54b559d8a4a5f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43323 } }
I20240430 17:04:24.475281 4192 leader_election.cc:288] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761), 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435)
W20240430 17:04:24.479401 3952 leader_election.cc:334] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1488849a12e4e3dae956cd218f4ed22 (127.0.103.65:38761): Network error: Client connection negotiation failed: client connection to 127.0.103.65:38761: connect: Connection refused (error 111)
W20240430 17:04:24.480562 3953 leader_election.cc:334] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5e8e00d436a14372a8280644f779828f (127.0.103.66:36435): Network error: Client connection negotiation failed: client connection to 127.0.103.66:36435: connect: Connection refused (error 111)
I20240430 17:04:24.480896 3953 leader_election.cc:302] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64b1d8e536e141a89a54b559d8a4a5f9; no voters: 5e8e00d436a14372a8280644f779828f, f1488849a12e4e3dae956cd218f4ed22
I20240430 17:04:24.481485 4192 raft_consensus.cc:2726] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 17:04:24.529681 413 tablet_server.cc:194] TabletServer@127.0.103.66:0 shutdown complete.
I20240430 17:04:24.543579 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20240430 17:04:24.565536 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:04:24.566144 413 tablet_replica.cc:330] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9: stopping tablet replica
I20240430 17:04:24.566709 413 raft_consensus.cc:2229] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:04:24.567525 413 raft_consensus.cc:2258] T 4a8fe0147cfb4fbf897f6644d81746c0 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:24.569510 413 tablet_replica.cc:330] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9: stopping tablet replica
I20240430 17:04:24.569949 413 raft_consensus.cc:2229] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:04:24.570358 413 raft_consensus.cc:2258] T 7dbf9f062d8d4528889226fa1567dfd6 P 64b1d8e536e141a89a54b559d8a4a5f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:24.594491 413 tablet_server.cc:194] TabletServer@127.0.103.67:0 shutdown complete.
I20240430 17:04:24.607754 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20240430 17:04:24.628355 413 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240430 17:04:24.628955 413 tablet_replica.cc:330] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b: stopping tablet replica
I20240430 17:04:24.629540 413 raft_consensus.cc:2229] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 17:04:24.630067 413 raft_consensus.cc:2258] T 4a8fe0147cfb4fbf897f6644d81746c0 P 3fd7cfcf66b640158a1b2f30112ca89b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:24.631989 413 tablet_replica.cc:330] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b: stopping tablet replica
I20240430 17:04:24.632433 413 raft_consensus.cc:2229] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 17:04:24.633206 413 raft_consensus.cc:2258] T 6fcb848e4c1b4360a8cb22f4418a4afd P 3fd7cfcf66b640158a1b2f30112ca89b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:24.653208 413 tablet_server.cc:194] TabletServer@127.0.103.68:0 shutdown complete.
I20240430 17:04:24.663676 413 master.cc:537] Master@127.0.103.126:37467 shutting down...
I20240430 17:04:24.680754 413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [term 1 LEADER]: Raft consensus shutting down.
I20240430 17:04:24.681377 413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 17:04:24.681695 413 tablet_replica.cc:330] T 00000000000000000000000000000000 P 36c59c6b68124a2cab9bdbd105645dd1: stopping tablet replica
I20240430 17:04:24.699456 413 master.cc:552] Master@127.0.103.126:37467 shutdown complete.
*** Aborted at 1714496664 (unix time) try "date -d @1714496664" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 413 (TID 0x7f79d02f9700) from PID 0; stack trace: ***
@ 0x2e41a0 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@3/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e6683 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@3/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f7a0726d980 (unknown) at ??:0
@ 0x7f7a0ce3b8fd std::__1::__hash_table<>::find<>() at ??:0
@ 0x7f7a06930bd1 std::__1::unordered_set<>::find() at ??:0
@ 0x7f7a069187c5 ContainsKey<>() at ??:0
@ 0x7f7a03f05388 kudu::RWMutex::CheckLockState() at ??:0
@ 0x7f7a03f058f3 kudu::RWMutex::TryReadLock() at ??:0
@ 0x7f7a0ce5850a kudu::RWMutex::try_lock_shared() at ??:0
@ 0x7f7a0cdf09b7 std::__1::shared_lock<>::shared_lock() at ??:0
@ 0x7f7a0cdc19d9 kudu::master::CatalogManager::ScopedLeaderSharedLock::ScopedLeaderSharedLock() at ??:0
@ 0x7f7a0cd8066f kudu::master::AutoLeaderRebalancerTask::RunLeaderRebalancer() at ??:0
@ 0x7f7a0cd80c5a kudu::master::AutoLeaderRebalancerTask::RunLoop() at ??:0
@ 0x7f7a0cd81e92 kudu::master::AutoLeaderRebalancerTask::Init()::$_0::operator()() at ??:0
@ 0x7f7a0cd81e4a _ZNSt3__18__invokeIRZN4kudu6master24AutoLeaderRebalancerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f7a0cd81dda std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f7a0cd81da2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f7a0cd8109e std::__1::__function::__func<>::operator()() at ??:0
@ 0x7f7a0d1b1da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7f7a0d1b1bda std::__1::function<>::operator()() at ??:0
@ 0x7f7a03f1f937 kudu::Thread::SuperviseThread() at ??:0
@ 0x2e4060 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@3/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7f7a072626db start_thread at ??:0
@ 0x7f7a008c471f clone at ??:0