[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 14:03:56.674378 13945 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.13.158.126:46667,127.13.158.125:44479,127.13.158.124:35541
I20240503 14:03:56.676012 13945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:03:56.677003 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:03:56.692479 13951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:56.693075 13954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:56.693547 13952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:56.693853 13945 server_base.cc:1034] running on GCE node
I20240503 14:03:57.749825 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:03:57.750078 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:03:57.750236 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745037750220 us; error 0 us; skew 500 ppm
I20240503 14:03:57.751101 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:03:57.756501 13945 webserver.cc:460] Webserver started at http://127.13.158.126:46251/ using document root <none> and password file <none>
I20240503 14:03:57.757329 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:03:57.757566 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:03:57.757972 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:03:57.764425 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-0-root/instance:
uuid: "98d36326b0ad4a76b1f03e253c781ce1"
format_stamp: "Formatted at 2024-05-03 14:03:57 on dist-test-slave-jm3r"
I20240503 14:03:57.772315 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20240503 14:03:57.777473 13960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:57.778908 13945 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:03:57.779412 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-0-root
uuid: "98d36326b0ad4a76b1f03e253c781ce1"
format_stamp: "Formatted at 2024-05-03 14:03:57 on dist-test-slave-jm3r"
I20240503 14:03:57.779883 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:03:57.840546 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:03:57.842082 13945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:03:57.842597 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:03:57.910854 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.126:46667
I20240503 14:03:57.910934 14011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.126:46667 every 8 connection(s)
I20240503 14:03:57.916886 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:03:57.918357 14012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:03:57.924127 14014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:57.925333 14015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:57.928732 13945 server_base.cc:1034] running on GCE node
W20240503 14:03:57.929162 14017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:57.930019 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:03:57.930217 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:03:57.930395 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745037930375 us; error 0 us; skew 500 ppm
I20240503 14:03:57.930994 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:03:57.935142 13945 webserver.cc:460] Webserver started at http://127.13.158.125:44577/ using document root <none> and password file <none>
I20240503 14:03:57.935644 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:03:57.935843 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:03:57.936094 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:03:57.937150 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-1-root/instance:
uuid: "828e04580a0148c3a762014f7c695bbb"
format_stamp: "Formatted at 2024-05-03 14:03:57 on dist-test-slave-jm3r"
I20240503 14:03:57.942554 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:03:57.941378 14012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:03:57.947198 14022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:57.948171 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:03:57.948525 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-1-root
uuid: "828e04580a0148c3a762014f7c695bbb"
format_stamp: "Formatted at 2024-05-03 14:03:57 on dist-test-slave-jm3r"
I20240503 14:03:57.948938 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:03:57.970667 14012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:03:57.973003 13961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.158.125:44479: connect: Connection refused (error 111)
I20240503 14:03:57.974241 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:03:57.975937 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240503 14:03:57.976990 14012 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.13.158.125:44479: Network error: Client connection negotiation failed: client connection to 127.13.158.125:44479: connect: Connection refused (error 111)
I20240503 14:03:58.026523 14012 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } attempt: 1
W20240503 14:03:58.030977 14012 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.13.158.125:44479: Network error: Client connection negotiation failed: client connection to 127.13.158.125:44479: connect: Connection refused (error 111)
I20240503 14:03:58.040946 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.125:44479
I20240503 14:03:58.041112 14076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.125:44479 every 8 connection(s)
I20240503 14:03:58.045557 14077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:58.046201 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:03:58.051815 14077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:03:58.053229 14079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:58.055342 14080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:58.060587 14082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:58.061033 13945 server_base.cc:1034] running on GCE node
I20240503 14:03:58.061798 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:03:58.062079 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:03:58.062273 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745038062255 us; error 0 us; skew 500 ppm
I20240503 14:03:58.062988 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:03:58.066610 13945 webserver.cc:460] Webserver started at http://127.13.158.124:35821/ using document root <none> and password file <none>
I20240503 14:03:58.067270 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:03:58.067534 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:03:58.067842 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:03:58.068207 14077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:03:58.069140 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-2-root/instance:
uuid: "33f10ab2b34345149ac1ff1024c4e254"
format_stamp: "Formatted at 2024-05-03 14:03:58 on dist-test-slave-jm3r"
I20240503 14:03:58.075656 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20240503 14:03:58.080601 14089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:58.080600 14077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:03:58.081616 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240503 14:03:58.081916 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-2-root
uuid: "33f10ab2b34345149ac1ff1024c4e254"
format_stamp: "Formatted at 2024-05-03 14:03:58 on dist-test-slave-jm3r"
I20240503 14:03:58.082188 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240503 14:03:58.085990 14077 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.13.158.124:35541: Network error: Client connection negotiation failed: client connection to 127.13.158.124:35541: connect: Connection refused (error 111)
I20240503 14:03:58.096545 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:03:58.097903 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:03:58.099871 14012 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } attempt: 2
I20240503 14:03:58.110391 14012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:03:58.115658 14012 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.13.158.124:35541: Network error: Client connection negotiation failed: client connection to 127.13.158.124:35541: connect: Connection refused (error 111)
I20240503 14:03:58.129578 14077 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } attempt: 1
W20240503 14:03:58.135183 14077 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.13.158.124:35541: Network error: Client connection negotiation failed: client connection to 127.13.158.124:35541: connect: Connection refused (error 111)
I20240503 14:03:58.147397 14012 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } attempt: 1
W20240503 14:03:58.152354 14012 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.13.158.124:35541: Network error: Client connection negotiation failed: client connection to 127.13.158.124:35541: connect: Connection refused (error 111)
I20240503 14:03:58.166687 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.124:35541
I20240503 14:03:58.166826 14141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.124:35541 every 8 connection(s)
I20240503 14:03:58.170609 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:03:58.171571 14142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:58.176787 14142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:03:58.188046 14142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:03:58.196691 14142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:03:58.210706 14077 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } attempt: 2
I20240503 14:03:58.216683 14142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254: Bootstrap starting.
I20240503 14:03:58.219882 14012 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } attempt: 2
I20240503 14:03:58.224558 14142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:58.227191 14142 log.cc:824] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254: Log is configured to *not* fsync() on all Append() calls
I20240503 14:03:58.228229 14077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb: Bootstrap starting.
I20240503 14:03:58.233812 14142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254: No bootstrap required, opened a new log
I20240503 14:03:58.234321 14077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:58.234463 14012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1: Bootstrap starting.
I20240503 14:03:58.239089 14012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:58.240578 14077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb: No bootstrap required, opened a new log
I20240503 14:03:58.244313 14012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1: No bootstrap required, opened a new log
I20240503 14:03:58.257623 14012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } }
I20240503 14:03:58.257743 14142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } }
I20240503 14:03:58.257630 14077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } }
I20240503 14:03:58.258641 14142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:58.258652 14012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:58.258711 14077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:58.259112 14012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d36326b0ad4a76b1f03e253c781ce1, State: Initialized, Role: FOLLOWER
I20240503 14:03:58.259157 14077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 828e04580a0148c3a762014f7c695bbb, State: Initialized, Role: FOLLOWER
I20240503 14:03:58.259083 14142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f10ab2b34345149ac1ff1024c4e254, State: Initialized, Role: FOLLOWER
I20240503 14:03:58.260237 14012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } }
I20240503 14:03:58.260246 14077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } }
I20240503 14:03:58.260246 14142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } }
I20240503 14:03:58.265038 14151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } } }
I20240503 14:03:58.265147 14149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } } }
I20240503 14:03:58.266207 14151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:03:58.266219 14149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:03:58.267221 14077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:03:58.269146 14142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:03:58.269639 14150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } } }
I20240503 14:03:58.270570 14150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:03:58.268913 14012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:03:58.310889 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240503 14:03:58.311249 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240503 14:03:58.316951 14182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:03:58.317339 14182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 14:03:58.317651 14178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:03:58.317988 14178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 14:03:58.319526 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:03:58.321372 14185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:03:58.321615 14185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 14:03:58.327442 14186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:58.328941 14187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:58.333140 14189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:58.333921 13945 server_base.cc:1034] running on GCE node
I20240503 14:03:58.334760 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:03:58.334995 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:03:58.335168 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745038335148 us; error 0 us; skew 500 ppm
I20240503 14:03:58.335743 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:03:58.338709 13945 webserver.cc:460] Webserver started at http://127.13.158.65:36053/ using document root <none> and password file <none>
I20240503 14:03:58.339161 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:03:58.339347 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:03:58.339607 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:03:58.340682 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-0-root/instance:
uuid: "0f08048640444a25a774f93a67c70c83"
format_stamp: "Formatted at 2024-05-03 14:03:58 on dist-test-slave-jm3r"
I20240503 14:03:58.345400 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
W20240503 14:03:58.347878 13966 tablet.cc:2369] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:03:58.348971 14194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:58.349781 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 14:03:58.350059 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-0-root
uuid: "0f08048640444a25a774f93a67c70c83"
format_stamp: "Formatted at 2024-05-03 14:03:58 on dist-test-slave-jm3r"
I20240503 14:03:58.350340 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:03:58.364727 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:03:58.366029 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:03:58.368451 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:03:58.373147 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:03:58.373385 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:58.373646 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:03:58.373795 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:58.466447 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.65:37475
I20240503 14:03:58.466547 14256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.65:37475 every 8 connection(s)
I20240503 14:03:58.491940 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:03:58.502890 14259 heartbeater.cc:347] Connected to a master server at 127.13.158.126:46667
I20240503 14:03:58.503403 14259 heartbeater.cc:464] Registering TS with master...
I20240503 14:03:58.504863 14259 heartbeater.cc:511] Master 127.13.158.126:46667 requested a full tablet report, sending...
W20240503 14:03:58.506659 14264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:58.507489 14258 heartbeater.cc:347] Connected to a master server at 127.13.158.125:44479
I20240503 14:03:58.507942 14258 heartbeater.cc:464] Registering TS with master...
I20240503 14:03:58.508324 14257 heartbeater.cc:347] Connected to a master server at 127.13.158.124:35541
I20240503 14:03:58.508692 14257 heartbeater.cc:464] Registering TS with master...
I20240503 14:03:58.508867 14258 heartbeater.cc:511] Master 127.13.158.125:44479 requested a full tablet report, sending...
I20240503 14:03:58.509567 14257 heartbeater.cc:511] Master 127.13.158.124:35541 requested a full tablet report, sending...
I20240503 14:03:58.513001 13977 ts_manager.cc:194] Registered new tserver with Master: 0f08048640444a25a774f93a67c70c83 (127.13.158.65:37475)
I20240503 14:03:58.516681 14042 ts_manager.cc:194] Registered new tserver with Master: 0f08048640444a25a774f93a67c70c83 (127.13.158.65:37475)
I20240503 14:03:58.520720 14107 ts_manager.cc:194] Registered new tserver with Master: 0f08048640444a25a774f93a67c70c83 (127.13.158.65:37475)
W20240503 14:03:58.522254 14265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:58.522975 13945 server_base.cc:1034] running on GCE node
W20240503 14:03:58.525306 14267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:58.526211 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:03:58.526494 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:03:58.526675 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745038526660 us; error 0 us; skew 500 ppm
I20240503 14:03:58.527204 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:03:58.529881 13945 webserver.cc:460] Webserver started at http://127.13.158.66:46685/ using document root <none> and password file <none>
I20240503 14:03:58.530349 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:03:58.530591 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:03:58.530882 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:03:58.531965 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-1-root/instance:
uuid: "6d4508b5d8664895ade57a5d39419efc"
format_stamp: "Formatted at 2024-05-03 14:03:58 on dist-test-slave-jm3r"
I20240503 14:03:58.536902 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:03:58.540463 14272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:58.541365 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:03:58.541653 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-1-root
uuid: "6d4508b5d8664895ade57a5d39419efc"
format_stamp: "Formatted at 2024-05-03 14:03:58 on dist-test-slave-jm3r"
I20240503 14:03:58.541949 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:03:58.566248 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:03:58.567536 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:03:58.569841 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:03:58.570590 14151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:58.571288 14151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } }
I20240503 14:03:58.573657 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:03:58.573937 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:58.574215 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:03:58.574457 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:58.575343 14151 leader_election.cc:288] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98d36326b0ad4a76b1f03e253c781ce1 (127.13.158.126:46667), 33f10ab2b34345149ac1ff1024c4e254 (127.13.158.124:35541)
I20240503 14:03:58.575459 14117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "828e04580a0148c3a762014f7c695bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f10ab2b34345149ac1ff1024c4e254" is_pre_election: true
I20240503 14:03:58.576164 13987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "828e04580a0148c3a762014f7c695bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d36326b0ad4a76b1f03e253c781ce1" is_pre_election: true
I20240503 14:03:58.576480 14117 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 828e04580a0148c3a762014f7c695bbb in term 0.
I20240503 14:03:58.576864 13987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 828e04580a0148c3a762014f7c695bbb in term 0.
I20240503 14:03:58.578253 14028 leader_election.cc:302] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33f10ab2b34345149ac1ff1024c4e254, 828e04580a0148c3a762014f7c695bbb; no voters:
I20240503 14:03:58.579445 14151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:03:58.579864 14151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:03:58.580219 14151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:58.588121 14151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } }
I20240503 14:03:58.591243 14151 leader_election.cc:288] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [CANDIDATE]: Term 1 election: Requested vote from peers 98d36326b0ad4a76b1f03e253c781ce1 (127.13.158.126:46667), 33f10ab2b34345149ac1ff1024c4e254 (127.13.158.124:35541)
I20240503 14:03:58.591549 13987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "828e04580a0148c3a762014f7c695bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d36326b0ad4a76b1f03e253c781ce1"
I20240503 14:03:58.592401 13987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:58.593632 14117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "828e04580a0148c3a762014f7c695bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f10ab2b34345149ac1ff1024c4e254"
I20240503 14:03:58.594241 14117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:58.600198 13987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 828e04580a0148c3a762014f7c695bbb in term 1.
I20240503 14:03:58.602223 14117 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 828e04580a0148c3a762014f7c695bbb in term 1.
I20240503 14:03:58.602465 14027 leader_election.cc:302] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 828e04580a0148c3a762014f7c695bbb, 98d36326b0ad4a76b1f03e253c781ce1; no voters:
I20240503 14:03:58.603717 14151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:03:58.606484 14151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 1 LEADER]: Becoming Leader. State: Replica: 828e04580a0148c3a762014f7c695bbb, State: Running, Role: LEADER
I20240503 14:03:58.607755 14151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } }
I20240503 14:03:58.622362 14282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 828e04580a0148c3a762014f7c695bbb. Latest consensus state: current_term: 1 leader_uuid: "828e04580a0148c3a762014f7c695bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } } }
I20240503 14:03:58.623448 14282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [sys.catalog]: This master's current role is: LEADER
I20240503 14:03:58.627202 14293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:03:58.635205 14293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:03:58.656728 14117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [term 1 FOLLOWER]: Refusing update from remote peer 828e04580a0148c3a762014f7c695bbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:03:58.656997 13987 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [term 1 FOLLOWER]: Refusing update from remote peer 828e04580a0148c3a762014f7c695bbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:03:58.658684 14282 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:03:58.660079 14151 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:03:58.688882 14150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 828e04580a0148c3a762014f7c695bbb. Latest consensus state: current_term: 1 leader_uuid: "828e04580a0148c3a762014f7c695bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } } }
I20240503 14:03:58.689756 14150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:03:58.692279 14149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 828e04580a0148c3a762014f7c695bbb. Latest consensus state: current_term: 1 leader_uuid: "828e04580a0148c3a762014f7c695bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } } }
I20240503 14:03:58.693106 14149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:03:58.698719 14302 mvcc.cc:205] Tried to move back new op lower bound from 7023595678312857600 to 7023595678150336512. Current Snapshot: MvccSnapshot[applied={T|T < 7023595678312857600}]
I20240503 14:03:58.700829 14282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "828e04580a0148c3a762014f7c695bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } } }
I20240503 14:03:58.702059 14282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [sys.catalog]: This master's current role is: LEADER
I20240503 14:03:58.702844 14282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "828e04580a0148c3a762014f7c695bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } } }
I20240503 14:03:58.703768 14282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [sys.catalog]: This master's current role is: LEADER
I20240503 14:03:58.704281 14293 catalog_manager.cc:1347] Generated new cluster ID: 5c0c3138db4b4faf992e63f7df22e70d
I20240503 14:03:58.704619 14293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:03:58.717118 14150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "828e04580a0148c3a762014f7c695bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } } }
I20240503 14:03:58.718603 14150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:03:58.722010 14149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "828e04580a0148c3a762014f7c695bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d36326b0ad4a76b1f03e253c781ce1" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 46667 } } peers { permanent_uuid: "828e04580a0148c3a762014f7c695bbb" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 44479 } } peers { permanent_uuid: "33f10ab2b34345149ac1ff1024c4e254" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 35541 } } }
I20240503 14:03:58.724472 14149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:03:58.770735 14293 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:03:58.773226 14293 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:03:58.819304 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.66:38747
I20240503 14:03:58.819378 14346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.66:38747 every 8 connection(s)
I20240503 14:03:58.821653 14293 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb: Generated new TSK 0
I20240503 14:03:58.822963 14293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:03:58.848891 14347 heartbeater.cc:347] Connected to a master server at 127.13.158.124:35541
I20240503 14:03:58.849382 14347 heartbeater.cc:464] Registering TS with master...
I20240503 14:03:58.850417 14347 heartbeater.cc:511] Master 127.13.158.124:35541 requested a full tablet report, sending...
I20240503 14:03:58.856309 14107 ts_manager.cc:194] Registered new tserver with Master: 6d4508b5d8664895ade57a5d39419efc (127.13.158.66:38747)
I20240503 14:03:58.867113 14348 heartbeater.cc:347] Connected to a master server at 127.13.158.125:44479
I20240503 14:03:58.867532 14348 heartbeater.cc:464] Registering TS with master...
I20240503 14:03:58.868525 14348 heartbeater.cc:511] Master 127.13.158.125:44479 requested a full tablet report, sending...
I20240503 14:03:58.871042 14042 ts_manager.cc:194] Registered new tserver with Master: 6d4508b5d8664895ade57a5d39419efc (127.13.158.66:38747)
I20240503 14:03:58.871531 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:03:58.874282 14042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55436
I20240503 14:03:58.877184 14350 heartbeater.cc:347] Connected to a master server at 127.13.158.126:46667
I20240503 14:03:58.877578 14350 heartbeater.cc:464] Registering TS with master...
I20240503 14:03:58.878334 14350 heartbeater.cc:511] Master 127.13.158.126:46667 requested a full tablet report, sending...
I20240503 14:03:58.881361 13977 ts_manager.cc:194] Registered new tserver with Master: 6d4508b5d8664895ade57a5d39419efc (127.13.158.66:38747)
W20240503 14:03:58.882414 14353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:58.884176 14354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:58.889353 14356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:58.889583 13945 server_base.cc:1034] running on GCE node
I20240503 14:03:58.891080 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:03:58.891376 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:03:58.891602 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745038891583 us; error 0 us; skew 500 ppm
I20240503 14:03:58.892343 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:03:58.895920 13945 webserver.cc:460] Webserver started at http://127.13.158.67:43945/ using document root <none> and password file <none>
I20240503 14:03:58.896554 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:03:58.896800 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:03:58.897140 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:03:58.898617 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-2-root/instance:
uuid: "1b4e9417d3ae4d14a2c467edcf588caf"
format_stamp: "Formatted at 2024-05-03 14:03:58 on dist-test-slave-jm3r"
I20240503 14:03:58.904660 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 14:03:58.909478 14361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:58.910395 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 14:03:58.910741 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-2-root
uuid: "1b4e9417d3ae4d14a2c467edcf588caf"
format_stamp: "Formatted at 2024-05-03 14:03:58 on dist-test-slave-jm3r"
I20240503 14:03:58.911036 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:03:58.940891 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:03:58.942144 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:03:58.944588 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:03:58.947664 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:03:58.947865 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:58.948132 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:03:58.948310 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:59.040524 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.67:42549
I20240503 14:03:59.040596 14423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.67:42549 every 8 connection(s)
I20240503 14:03:59.065623 14424 heartbeater.cc:347] Connected to a master server at 127.13.158.124:35541
I20240503 14:03:59.066120 14424 heartbeater.cc:464] Registering TS with master...
I20240503 14:03:59.067559 14424 heartbeater.cc:511] Master 127.13.158.124:35541 requested a full tablet report, sending...
I20240503 14:03:59.070818 14107 ts_manager.cc:194] Registered new tserver with Master: 1b4e9417d3ae4d14a2c467edcf588caf (127.13.158.67:42549)
I20240503 14:03:59.073344 14426 heartbeater.cc:347] Connected to a master server at 127.13.158.126:46667
I20240503 14:03:59.073750 14426 heartbeater.cc:464] Registering TS with master...
I20240503 14:03:59.074795 14426 heartbeater.cc:511] Master 127.13.158.126:46667 requested a full tablet report, sending...
I20240503 14:03:59.074790 14425 heartbeater.cc:347] Connected to a master server at 127.13.158.125:44479
I20240503 14:03:59.076035 14425 heartbeater.cc:464] Registering TS with master...
I20240503 14:03:59.077024 14425 heartbeater.cc:511] Master 127.13.158.125:44479 requested a full tablet report, sending...
I20240503 14:03:59.079090 14042 ts_manager.cc:194] Registered new tserver with Master: 1b4e9417d3ae4d14a2c467edcf588caf (127.13.158.67:42549)
I20240503 14:03:59.080484 14042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55444
I20240503 14:03:59.082592 13977 ts_manager.cc:194] Registered new tserver with Master: 1b4e9417d3ae4d14a2c467edcf588caf (127.13.158.67:42549)
I20240503 14:03:59.082788 13945 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.02488304s
I20240503 14:03:59.321576 14182 catalog_manager.cc:1259] Loaded cluster ID: 5c0c3138db4b4faf992e63f7df22e70d
I20240503 14:03:59.325120 14185 catalog_manager.cc:1259] Loaded cluster ID: 5c0c3138db4b4faf992e63f7df22e70d
I20240503 14:03:59.325381 14182 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1: loading cluster ID for follower catalog manager: success
I20240503 14:03:59.325568 14185 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254: loading cluster ID for follower catalog manager: success
I20240503 14:03:59.330122 14182 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1: acquiring CA information for follower catalog manager: success
I20240503 14:03:59.332515 14185 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254: acquiring CA information for follower catalog manager: success
I20240503 14:03:59.334203 14182 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 14:03:59.338089 14185 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 14:03:59.522325 14042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39556
I20240503 14:03:59.878191 14348 heartbeater.cc:503] Master 127.13.158.125:44479 was elected leader, sending a full tablet report...
I20240503 14:04:00.083618 14425 heartbeater.cc:503] Master 127.13.158.125:44479 was elected leader, sending a full tablet report...
I20240503 14:04:00.305130 14175 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:00.395520 13945 test_util.cc:268] Using random seed: 822303961
I20240503 14:04:00.460212 14042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55446:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:04:00.524966 14258 heartbeater.cc:503] Master 127.13.158.125:44479 was elected leader, sending a full tablet report...
I20240503 14:04:00.535859 14222 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9f44309a6f4602a28e24787c8df7ca (DEFAULT_TABLE table=test-workload [id=a324c0d67381435db5f2c878d630e34c]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 14:04:00.537619 14222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9f44309a6f4602a28e24787c8df7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:00.539517 14221 tablet_service.cc:1467] Processing CreateTablet for tablet eb17e6a0e45a4105a2b6adefd1e70e92 (DEFAULT_TABLE table=test-workload [id=a324c0d67381435db5f2c878d630e34c]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 14:04:00.540947 14221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb17e6a0e45a4105a2b6adefd1e70e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:00.549878 14304 tablet_service.cc:1467] Processing CreateTablet for tablet a7d3b967fbb14ca0ac464ed657569a8c (DEFAULT_TABLE table=test-workload [id=a324c0d67381435db5f2c878d630e34c]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 14:04:00.551453 14304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d3b967fbb14ca0ac464ed657569a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:00.551302 14389 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6726f3b6ec45faad0f19552a0e1bf9 (DEFAULT_TABLE table=test-workload [id=a324c0d67381435db5f2c878d630e34c]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 14:04:00.552654 14389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6726f3b6ec45faad0f19552a0e1bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:00.564987 14458 tablet_bootstrap.cc:492] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83: Bootstrap starting.
I20240503 14:04:00.572889 14460 tablet_bootstrap.cc:492] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf: Bootstrap starting.
I20240503 14:04:00.574913 14458 tablet_bootstrap.cc:654] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:00.575783 14459 tablet_bootstrap.cc:492] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc: Bootstrap starting.
I20240503 14:04:00.580966 14460 tablet_bootstrap.cc:654] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:00.582196 14459 tablet_bootstrap.cc:654] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:00.587568 14458 tablet_bootstrap.cc:492] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83: No bootstrap required, opened a new log
I20240503 14:04:00.587767 14460 tablet_bootstrap.cc:492] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf: No bootstrap required, opened a new log
I20240503 14:04:00.588145 14458 ts_tablet_manager.cc:1367] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.001s
I20240503 14:04:00.588348 14460 ts_tablet_manager.cc:1367] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20240503 14:04:00.590224 14459 tablet_bootstrap.cc:492] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc: No bootstrap required, opened a new log
I20240503 14:04:00.590723 14459 ts_tablet_manager.cc:1367] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240503 14:04:00.591498 14458 raft_consensus.cc:348] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f08048640444a25a774f93a67c70c83" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 37475 } }
I20240503 14:04:00.591733 14460 raft_consensus.cc:348] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4e9417d3ae4d14a2c467edcf588caf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 42549 } }
I20240503 14:04:00.592149 14458 raft_consensus.cc:374] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:00.592272 14460 raft_consensus.cc:374] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:00.592506 14458 raft_consensus.cc:729] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f08048640444a25a774f93a67c70c83, State: Initialized, Role: FOLLOWER
I20240503 14:04:00.592705 14460 raft_consensus.cc:729] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4e9417d3ae4d14a2c467edcf588caf, State: Initialized, Role: FOLLOWER
I20240503 14:04:00.593277 14458 consensus_queue.cc:260] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f08048640444a25a774f93a67c70c83" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 37475 } }
I20240503 14:04:00.593580 14459 raft_consensus.cc:348] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4508b5d8664895ade57a5d39419efc" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38747 } }
I20240503 14:04:00.594118 14458 raft_consensus.cc:388] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:00.593340 14460 consensus_queue.cc:260] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4e9417d3ae4d14a2c467edcf588caf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 42549 } }
I20240503 14:04:00.594199 14459 raft_consensus.cc:374] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:00.594578 14458 raft_consensus.cc:482] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:00.594760 14459 raft_consensus.cc:729] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4508b5d8664895ade57a5d39419efc, State: Initialized, Role: FOLLOWER
I20240503 14:04:00.594779 14460 raft_consensus.cc:388] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:00.595038 14458 raft_consensus.cc:3037] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:00.595477 14460 raft_consensus.cc:482] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:00.595935 14460 raft_consensus.cc:3037] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:00.595649 14459 consensus_queue.cc:260] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4508b5d8664895ade57a5d39419efc" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38747 } }
I20240503 14:04:00.596516 14459 raft_consensus.cc:388] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:00.596858 14459 raft_consensus.cc:482] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:00.597225 14459 raft_consensus.cc:3037] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:00.602165 14459 raft_consensus.cc:504] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4508b5d8664895ade57a5d39419efc" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38747 } }
I20240503 14:04:00.602545 14458 raft_consensus.cc:504] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f08048640444a25a774f93a67c70c83" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 37475 } }
I20240503 14:04:00.602859 14459 leader_election.cc:302] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d4508b5d8664895ade57a5d39419efc; no voters:
I20240503 14:04:00.603370 14458 leader_election.cc:302] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f08048640444a25a774f93a67c70c83; no voters:
I20240503 14:04:00.603545 14460 raft_consensus.cc:504] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4e9417d3ae4d14a2c467edcf588caf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 42549 } }
I20240503 14:04:00.604388 14460 leader_election.cc:302] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b4e9417d3ae4d14a2c467edcf588caf; no voters:
I20240503 14:04:00.605763 14459 leader_election.cc:288] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:00.606294 14464 raft_consensus.cc:2781] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:00.610100 14458 leader_election.cc:288] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:00.615226 14460 leader_election.cc:288] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:00.611673 14465 raft_consensus.cc:2781] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:00.623967 14466 raft_consensus.cc:2781] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:00.627583 14459 ts_tablet_manager.cc:1398] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc: Time spent starting tablet: real 0.037s user 0.013s sys 0.011s
I20240503 14:04:00.668265 14458 ts_tablet_manager.cc:1398] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83: Time spent starting tablet: real 0.080s user 0.028s sys 0.048s
I20240503 14:04:00.668326 14466 raft_consensus.cc:686] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [term 1 LEADER]: Becoming Leader. State: Replica: 1b4e9417d3ae4d14a2c467edcf588caf, State: Running, Role: LEADER
I20240503 14:04:00.672495 14464 raft_consensus.cc:686] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [term 1 LEADER]: Becoming Leader. State: Replica: 6d4508b5d8664895ade57a5d39419efc, State: Running, Role: LEADER
I20240503 14:04:00.673339 14460 ts_tablet_manager.cc:1398] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf: Time spent starting tablet: real 0.085s user 0.032s sys 0.036s
I20240503 14:04:00.670445 14466 consensus_queue.cc:237] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4e9417d3ae4d14a2c467edcf588caf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 42549 } }
I20240503 14:04:00.673370 14464 consensus_queue.cc:237] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4508b5d8664895ade57a5d39419efc" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38747 } }
I20240503 14:04:00.678403 14458 tablet_bootstrap.cc:492] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83: Bootstrap starting.
I20240503 14:04:00.679848 14465 raft_consensus.cc:686] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [term 1 LEADER]: Becoming Leader. State: Replica: 0f08048640444a25a774f93a67c70c83, State: Running, Role: LEADER
I20240503 14:04:00.680713 14465 consensus_queue.cc:237] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f08048640444a25a774f93a67c70c83" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 37475 } }
I20240503 14:04:00.686815 14458 tablet_bootstrap.cc:654] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:00.692760 14042 catalog_manager.cc:5484] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b4e9417d3ae4d14a2c467edcf588caf (127.13.158.67). New cstate: current_term: 1 leader_uuid: "1b4e9417d3ae4d14a2c467edcf588caf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4e9417d3ae4d14a2c467edcf588caf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 42549 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:00.698415 14041 catalog_manager.cc:5484] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4508b5d8664895ade57a5d39419efc (127.13.158.66). New cstate: current_term: 1 leader_uuid: "6d4508b5d8664895ade57a5d39419efc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4508b5d8664895ade57a5d39419efc" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38747 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:00.760848 14041 catalog_manager.cc:5484] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f08048640444a25a774f93a67c70c83 (127.13.158.65). New cstate: current_term: 1 leader_uuid: "0f08048640444a25a774f93a67c70c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f08048640444a25a774f93a67c70c83" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 37475 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:00.780444 14458 tablet_bootstrap.cc:492] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83: No bootstrap required, opened a new log
I20240503 14:04:00.781059 14458 ts_tablet_manager.cc:1367] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83: Time spent bootstrapping tablet: real 0.103s user 0.023s sys 0.043s
I20240503 14:04:00.785030 14458 raft_consensus.cc:348] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f08048640444a25a774f93a67c70c83" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 37475 } }
I20240503 14:04:00.785791 14458 raft_consensus.cc:374] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:00.786092 14458 raft_consensus.cc:729] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f08048640444a25a774f93a67c70c83, State: Initialized, Role: FOLLOWER
I20240503 14:04:00.787026 14458 consensus_queue.cc:260] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f08048640444a25a774f93a67c70c83" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 37475 } }
I20240503 14:04:00.787819 14458 raft_consensus.cc:388] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:00.788144 14458 raft_consensus.cc:482] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:00.788481 14458 raft_consensus.cc:3037] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:00.796260 14458 raft_consensus.cc:504] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f08048640444a25a774f93a67c70c83" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 37475 } }
I20240503 14:04:00.797261 14458 leader_election.cc:302] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f08048640444a25a774f93a67c70c83; no voters:
I20240503 14:04:00.798058 14458 leader_election.cc:288] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:00.798329 14472 raft_consensus.cc:2781] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:00.799875 14458 ts_tablet_manager.cc:1398] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20240503 14:04:00.801450 14472 raft_consensus.cc:686] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [term 1 LEADER]: Becoming Leader. State: Replica: 0f08048640444a25a774f93a67c70c83, State: Running, Role: LEADER
I20240503 14:04:00.802280 14472 consensus_queue.cc:237] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f08048640444a25a774f93a67c70c83" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 37475 } }
I20240503 14:04:00.827301 14041 catalog_manager.cc:5484] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f08048640444a25a774f93a67c70c83 (127.13.158.65). New cstate: current_term: 1 leader_uuid: "0f08048640444a25a774f93a67c70c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f08048640444a25a774f93a67c70c83" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 37475 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:00.877826 13945 tablet_server.cc:178] TabletServer@127.13.158.65:0 shutting down...
I20240503 14:04:00.903560 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:00.904759 13945 tablet_replica.cc:330] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83: stopping tablet replica
I20240503 14:04:00.905676 13945 raft_consensus.cc:2229] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:00.906180 13945 raft_consensus.cc:2258] T eb17e6a0e45a4105a2b6adefd1e70e92 P 0f08048640444a25a774f93a67c70c83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:00.909158 13945 tablet_replica.cc:330] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83: stopping tablet replica
I20240503 14:04:00.909677 13945 raft_consensus.cc:2229] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:00.910076 13945 raft_consensus.cc:2258] T 1f9f44309a6f4602a28e24787c8df7ca P 0f08048640444a25a774f93a67c70c83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:01.065536 13945 tablet_server.cc:194] TabletServer@127.13.158.65:0 shutdown complete.
I20240503 14:04:01.075474 13945 tablet_server.cc:178] TabletServer@127.13.158.66:0 shutting down...
I20240503 14:04:01.093307 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:01.094161 13945 tablet_replica.cc:330] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc: stopping tablet replica
I20240503 14:04:01.094707 13945 raft_consensus.cc:2229] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:01.095129 13945 raft_consensus.cc:2258] T a7d3b967fbb14ca0ac464ed657569a8c P 6d4508b5d8664895ade57a5d39419efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:01.115120 13945 tablet_server.cc:194] TabletServer@127.13.158.66:0 shutdown complete.
I20240503 14:04:01.123049 13945 tablet_server.cc:178] TabletServer@127.13.158.67:0 shutting down...
I20240503 14:04:01.139101 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:01.139992 13945 tablet_replica.cc:330] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf: stopping tablet replica
I20240503 14:04:01.140527 13945 raft_consensus.cc:2229] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:01.140941 13945 raft_consensus.cc:2258] T 2e6726f3b6ec45faad0f19552a0e1bf9 P 1b4e9417d3ae4d14a2c467edcf588caf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:01.160805 13945 tablet_server.cc:194] TabletServer@127.13.158.67:0 shutdown complete.
I20240503 14:04:01.169982 13945 master.cc:537] Master@127.13.158.126:46667 shutting down...
I20240503 14:04:01.278479 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:01.279004 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:01.279347 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 98d36326b0ad4a76b1f03e253c781ce1: stopping tablet replica
I20240503 14:04:01.299978 13945 master.cc:552] Master@127.13.158.126:46667 shutdown complete.
I20240503 14:04:01.311272 13945 master.cc:537] Master@127.13.158.125:44479 shutting down...
I20240503 14:04:01.324944 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:01.325789 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:01.326120 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 828e04580a0148c3a762014f7c695bbb: stopping tablet replica
I20240503 14:04:01.367468 13945 master.cc:552] Master@127.13.158.125:44479 shutdown complete.
I20240503 14:04:01.379436 13945 master.cc:537] Master@127.13.158.124:35541 shutting down...
I20240503 14:04:01.393968 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:01.394498 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:01.394799 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 33f10ab2b34345149ac1ff1024c4e254: stopping tablet replica
I20240503 14:04:01.412693 13945 master.cc:552] Master@127.13.158.124:35541 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (4788 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240503 14:04:01.459503 13945 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.13.158.126:39965
I20240503 14:04:01.460505 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:01.465590 14484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:01.466369 14485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:01.467489 14487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:01.468474 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:01.469190 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:01.469388 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:01.469537 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745041469518 us; error 0 us; skew 500 ppm
I20240503 14:04:01.470008 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:01.478742 13945 webserver.cc:460] Webserver started at http://127.13.158.126:46025/ using document root <none> and password file <none>
I20240503 14:04:01.479233 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:01.479420 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:01.479669 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:01.480785 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/master-0-root/instance:
uuid: "90b7767a39aa4fa09513935a8323e975"
format_stamp: "Formatted at 2024-05-03 14:04:01 on dist-test-slave-jm3r"
I20240503 14:04:01.485718 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:04:01.489457 14492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:01.490316 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:04:01.490733 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/master-0-root
uuid: "90b7767a39aa4fa09513935a8323e975"
format_stamp: "Formatted at 2024-05-03 14:04:01 on dist-test-slave-jm3r"
I20240503 14:04:01.491165 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:01.523193 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:01.524216 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:01.557974 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.126:39965
I20240503 14:04:01.558094 14543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.126:39965 every 8 connection(s)
I20240503 14:04:01.561497 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:01.571785 14544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975: Bootstrap starting.
I20240503 14:04:01.575459 14544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:01.579165 14544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975: No bootstrap required, opened a new log
I20240503 14:04:01.580914 14544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b7767a39aa4fa09513935a8323e975" member_type: VOTER }
I20240503 14:04:01.581303 14544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:01.581548 14544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b7767a39aa4fa09513935a8323e975, State: Initialized, Role: FOLLOWER
I20240503 14:04:01.582186 14544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b7767a39aa4fa09513935a8323e975" member_type: VOTER }
I20240503 14:04:01.582686 14544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:01.582903 14544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:01.583148 14544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:01.587168 14544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b7767a39aa4fa09513935a8323e975" member_type: VOTER }
I20240503 14:04:01.587695 14544 leader_election.cc:302] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90b7767a39aa4fa09513935a8323e975; no voters:
I20240503 14:04:01.588775 14544 leader_election.cc:288] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:01.589080 14547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:01.590277 14547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [term 1 LEADER]: Becoming Leader. State: Replica: 90b7767a39aa4fa09513935a8323e975, State: Running, Role: LEADER
I20240503 14:04:01.590966 14547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b7767a39aa4fa09513935a8323e975" member_type: VOTER }
I20240503 14:04:01.591533 14544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:04:01.594816 14548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90b7767a39aa4fa09513935a8323e975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b7767a39aa4fa09513935a8323e975" member_type: VOTER } }
I20240503 14:04:01.595043 14549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b7767a39aa4fa09513935a8323e975. Latest consensus state: current_term: 1 leader_uuid: "90b7767a39aa4fa09513935a8323e975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b7767a39aa4fa09513935a8323e975" member_type: VOTER } }
I20240503 14:04:01.595677 14548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:01.595690 14549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:01.598031 14554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:04:01.604888 14554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:04:01.607059 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:04:01.613432 14554 catalog_manager.cc:1347] Generated new cluster ID: 15699f75457041bab64289a859d85767
I20240503 14:04:01.613750 14554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:04:01.638368 14554 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:04:01.639621 14554 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:04:01.655329 14554 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975: Generated new TSK 0
I20240503 14:04:01.655969 14554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:04:01.673452 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:01.680344 14565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:01.681421 14566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:01.683321 14568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:01.685612 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:01.686475 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:01.686668 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:01.686797 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745041686785 us; error 0 us; skew 500 ppm
I20240503 14:04:01.687456 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:01.690188 13945 webserver.cc:460] Webserver started at http://127.13.158.65:35105/ using document root <none> and password file <none>
I20240503 14:04:01.690688 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:01.690846 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:01.691074 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:01.692255 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-0-root/instance:
uuid: "186b18bf626043b39784c11cebd55db0"
format_stamp: "Formatted at 2024-05-03 14:04:01 on dist-test-slave-jm3r"
I20240503 14:04:01.696820 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240503 14:04:01.699980 14573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:01.700795 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 14:04:01.701036 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-0-root
uuid: "186b18bf626043b39784c11cebd55db0"
format_stamp: "Formatted at 2024-05-03 14:04:01 on dist-test-slave-jm3r"
I20240503 14:04:01.701288 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:01.712430 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:01.713410 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:01.715092 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:01.717295 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:01.717486 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:01.717705 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:01.717859 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:01.754082 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.65:44101
I20240503 14:04:01.754180 14635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.65:44101 every 8 connection(s)
I20240503 14:04:01.758822 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:01.766310 14641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:01.767144 14640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:01.769032 13945 server_base.cc:1034] running on GCE node
W20240503 14:04:01.769467 14643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:01.770151 14636 heartbeater.cc:347] Connected to a master server at 127.13.158.126:39965
I20240503 14:04:01.770267 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 14:04:01.770499 14636 heartbeater.cc:464] Registering TS with master...
W20240503 14:04:01.770582 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:01.770853 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745041770837 us; error 0 us; skew 500 ppm
I20240503 14:04:01.771373 14636 heartbeater.cc:511] Master 127.13.158.126:39965 requested a full tablet report, sending...
I20240503 14:04:01.771464 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:01.773417 14509 ts_manager.cc:194] Registered new tserver with Master: 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101)
I20240503 14:04:01.774128 13945 webserver.cc:460] Webserver started at http://127.13.158.66:44901/ using document root <none> and password file <none>
I20240503 14:04:01.774649 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:01.774873 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:01.775116 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:01.775348 14509 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44558
I20240503 14:04:01.776237 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-1-root/instance:
uuid: "a7638e8e636a4601b99a740dd28fa895"
format_stamp: "Formatted at 2024-05-03 14:04:01 on dist-test-slave-jm3r"
I20240503 14:04:01.780588 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:04:01.783670 14648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:01.784523 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:04:01.784817 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-1-root
uuid: "a7638e8e636a4601b99a740dd28fa895"
format_stamp: "Formatted at 2024-05-03 14:04:01 on dist-test-slave-jm3r"
I20240503 14:04:01.785096 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:01.802381 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:01.803467 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:01.804850 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:01.807068 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:01.807264 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:01.807484 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:01.807631 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:01.845870 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.66:41235
I20240503 14:04:01.845897 14710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.66:41235 every 8 connection(s)
I20240503 14:04:01.851063 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:01.858042 14714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:01.859819 14715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:01.861158 14717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:01.861929 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:01.862782 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 14:04:01.862856 14711 heartbeater.cc:347] Connected to a master server at 127.13.158.126:39965
W20240503 14:04:01.863051 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:01.863268 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745041863251 us; error 0 us; skew 500 ppm
I20240503 14:04:01.863281 14711 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:01.863884 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:01.864100 14711 heartbeater.cc:511] Master 127.13.158.126:39965 requested a full tablet report, sending...
I20240503 14:04:01.866114 14509 ts_manager.cc:194] Registered new tserver with Master: a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:01.866578 13945 webserver.cc:460] Webserver started at http://127.13.158.67:39215/ using document root <none> and password file <none>
I20240503 14:04:01.867063 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:01.867244 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:01.867484 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:01.867551 14509 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44568
I20240503 14:04:01.868711 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-2-root/instance:
uuid: "36443f8bb0a844a7947756620254c2e0"
format_stamp: "Formatted at 2024-05-03 14:04:01 on dist-test-slave-jm3r"
I20240503 14:04:01.873476 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:04:01.877256 14722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:01.878088 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:04:01.878368 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-2-root
uuid: "36443f8bb0a844a7947756620254c2e0"
format_stamp: "Formatted at 2024-05-03 14:04:01 on dist-test-slave-jm3r"
I20240503 14:04:01.878718 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:01.888720 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:01.889979 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:01.892184 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:01.895056 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:01.895288 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:01.895530 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:01.895681 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:01.941794 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.67:45667
I20240503 14:04:01.941880 14784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.67:45667 every 8 connection(s)
I20240503 14:04:01.955258 14785 heartbeater.cc:347] Connected to a master server at 127.13.158.126:39965
I20240503 14:04:01.955672 14785 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:01.956470 14785 heartbeater.cc:511] Master 127.13.158.126:39965 requested a full tablet report, sending...
I20240503 14:04:01.958492 14509 ts_manager.cc:194] Registered new tserver with Master: 36443f8bb0a844a7947756620254c2e0 (127.13.158.67:45667)
I20240503 14:04:01.958777 13945 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013962663s
I20240503 14:04:01.960327 14509 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44572
I20240503 14:04:02.778086 14636 heartbeater.cc:503] Master 127.13.158.126:39965 was elected leader, sending a full tablet report...
I20240503 14:04:02.870079 14711 heartbeater.cc:503] Master 127.13.158.126:39965 was elected leader, sending a full tablet report...
I20240503 14:04:02.963091 14785 heartbeater.cc:503] Master 127.13.158.126:39965 was elected leader, sending a full tablet report...
I20240503 14:04:02.993193 13945 test_util.cc:268] Using random seed: 824901649
I20240503 14:04:03.015973 14509 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44584:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:04:03.018290 14509 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:04:03.082345 14747 tablet_service.cc:1467] Processing CreateTablet for tablet d2302ac8c2e544c68cf11427f3bf9143 (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 14:04:03.083426 14749 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3fa6230957409db8eff9dccbf62b32 (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 14:04:03.082347 14750 tablet_service.cc:1467] Processing CreateTablet for tablet b2a8632a21724f14ab7ddd5dcee692df (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 14:04:03.085336 14748 tablet_service.cc:1467] Processing CreateTablet for tablet 99f1c0f5f2354dacbd45ac12c93e70bc (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 14:04:03.083846 14747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2302ac8c2e544c68cf11427f3bf9143. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.087177 14746 tablet_service.cc:1467] Processing CreateTablet for tablet f023e4f1e70c47caad92f740e24b9acc (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 14:04:03.088457 14746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f023e4f1e70c47caad92f740e24b9acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.090934 14748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f1c0f5f2354dacbd45ac12c93e70bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.091713 14745 tablet_service.cc:1467] Processing CreateTablet for tablet 7759c34f658a42c5a42e9f8ca831fe07 (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 14:04:03.093351 14745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7759c34f658a42c5a42e9f8ca831fe07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.095034 14749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3fa6230957409db8eff9dccbf62b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.096838 14750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a8632a21724f14ab7ddd5dcee692df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.099963 14676 tablet_service.cc:1467] Processing CreateTablet for tablet b2a8632a21724f14ab7ddd5dcee692df (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 14:04:03.101311 14676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a8632a21724f14ab7ddd5dcee692df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.106518 14675 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3fa6230957409db8eff9dccbf62b32 (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 14:04:03.107946 14675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3fa6230957409db8eff9dccbf62b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.111759 14674 tablet_service.cc:1467] Processing CreateTablet for tablet 99f1c0f5f2354dacbd45ac12c93e70bc (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 14:04:03.113157 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f1c0f5f2354dacbd45ac12c93e70bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.115648 14673 tablet_service.cc:1467] Processing CreateTablet for tablet d2302ac8c2e544c68cf11427f3bf9143 (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 14:04:03.117058 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2302ac8c2e544c68cf11427f3bf9143. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.117599 14671 tablet_service.cc:1467] Processing CreateTablet for tablet 7759c34f658a42c5a42e9f8ca831fe07 (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 14:04:03.117722 14744 tablet_service.cc:1467] Processing CreateTablet for tablet 17937dd0f02348718b2704f4cddb0397 (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 14:04:03.118924 14671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7759c34f658a42c5a42e9f8ca831fe07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.111766 14672 tablet_service.cc:1467] Processing CreateTablet for tablet f023e4f1e70c47caad92f740e24b9acc (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 14:04:03.119285 14744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17937dd0f02348718b2704f4cddb0397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.119920 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f023e4f1e70c47caad92f740e24b9acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.121603 14670 tablet_service.cc:1467] Processing CreateTablet for tablet 17937dd0f02348718b2704f4cddb0397 (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 14:04:03.123078 14670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17937dd0f02348718b2704f4cddb0397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.132246 14601 tablet_service.cc:1467] Processing CreateTablet for tablet b2a8632a21724f14ab7ddd5dcee692df (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 14:04:03.133700 14601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a8632a21724f14ab7ddd5dcee692df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.134479 14600 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3fa6230957409db8eff9dccbf62b32 (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 14:04:03.136554 14599 tablet_service.cc:1467] Processing CreateTablet for tablet 99f1c0f5f2354dacbd45ac12c93e70bc (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 14:04:03.137928 14599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f1c0f5f2354dacbd45ac12c93e70bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.140583 14595 tablet_service.cc:1467] Processing CreateTablet for tablet 17937dd0f02348718b2704f4cddb0397 (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 14:04:03.143033 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3fa6230957409db8eff9dccbf62b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.143699 14595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17937dd0f02348718b2704f4cddb0397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.143390 14596 tablet_service.cc:1467] Processing CreateTablet for tablet 7759c34f658a42c5a42e9f8ca831fe07 (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 14:04:03.149140 14743 tablet_service.cc:1467] Processing CreateTablet for tablet 0dfef82a71c8452f99024d7e7edc988f (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 14:04:03.149788 14597 tablet_service.cc:1467] Processing CreateTablet for tablet f023e4f1e70c47caad92f740e24b9acc (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 14:04:03.151484 14743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dfef82a71c8452f99024d7e7edc988f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.152048 14596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7759c34f658a42c5a42e9f8ca831fe07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.145630 14598 tablet_service.cc:1467] Processing CreateTablet for tablet d2302ac8c2e544c68cf11427f3bf9143 (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 14:04:03.156204 14597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f023e4f1e70c47caad92f740e24b9acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.158550 14594 tablet_service.cc:1467] Processing CreateTablet for tablet 0dfef82a71c8452f99024d7e7edc988f (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 14:04:03.160481 14669 tablet_service.cc:1467] Processing CreateTablet for tablet 0dfef82a71c8452f99024d7e7edc988f (DEFAULT_TABLE table=test-workload [id=52fd6a1d78684d75914efb11d6bae343]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 14:04:03.162513 14669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dfef82a71c8452f99024d7e7edc988f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.166620 14598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2302ac8c2e544c68cf11427f3bf9143. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.167938 14594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dfef82a71c8452f99024d7e7edc988f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:03.222316 14805 tablet_bootstrap.cc:492] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0: Bootstrap starting.
I20240503 14:04:03.230285 14805 tablet_bootstrap.cc:654] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.251197 14807 tablet_bootstrap.cc:492] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895: Bootstrap starting.
I20240503 14:04:03.252836 14805 tablet_bootstrap.cc:492] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0: No bootstrap required, opened a new log
I20240503 14:04:03.253890 14805 ts_tablet_manager.cc:1367] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.005s
I20240503 14:04:03.257571 14805 raft_consensus.cc:348] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.258873 14808 tablet_bootstrap.cc:492] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0: Bootstrap starting.
I20240503 14:04:03.258900 14805 raft_consensus.cc:374] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.265194 14805 raft_consensus.cc:729] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.266124 14805 consensus_queue.cc:260] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.273152 14808 tablet_bootstrap.cc:654] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.273967 14807 tablet_bootstrap.cc:654] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.278613 14805 ts_tablet_manager.cc:1398] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0: Time spent starting tablet: real 0.024s user 0.013s sys 0.004s
I20240503 14:04:03.279796 14805 tablet_bootstrap.cc:492] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0: Bootstrap starting.
I20240503 14:04:03.285463 14805 tablet_bootstrap.cc:654] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.288540 14808 tablet_bootstrap.cc:492] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0: No bootstrap required, opened a new log
I20240503 14:04:03.289099 14808 ts_tablet_manager.cc:1367] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.000s
I20240503 14:04:03.290995 14807 tablet_bootstrap.cc:492] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895: No bootstrap required, opened a new log
I20240503 14:04:03.291733 14807 ts_tablet_manager.cc:1367] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.006s
I20240503 14:04:03.291594 14808 raft_consensus.cc:348] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.292619 14808 raft_consensus.cc:374] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.292923 14808 raft_consensus.cc:729] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b18bf626043b39784c11cebd55db0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.293609 14808 consensus_queue.cc:260] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.294785 14807 raft_consensus.cc:348] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.295459 14807 raft_consensus.cc:374] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.295691 14807 raft_consensus.cc:729] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7638e8e636a4601b99a740dd28fa895, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.296357 14807 consensus_queue.cc:260] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.297524 14805 tablet_bootstrap.cc:492] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0: No bootstrap required, opened a new log
I20240503 14:04:03.298074 14805 ts_tablet_manager.cc:1367] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.011s
I20240503 14:04:03.299383 14807 ts_tablet_manager.cc:1398] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:04:03.300542 14807 tablet_bootstrap.cc:492] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895: Bootstrap starting.
I20240503 14:04:03.300650 14805 raft_consensus.cc:348] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.301496 14805 raft_consensus.cc:374] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.301879 14805 raft_consensus.cc:729] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.302747 14805 consensus_queue.cc:260] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.305281 14805 ts_tablet_manager.cc:1398] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:04:03.306211 14805 tablet_bootstrap.cc:492] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0: Bootstrap starting.
I20240503 14:04:03.306685 14807 tablet_bootstrap.cc:654] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.302817 14808 ts_tablet_manager.cc:1398] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20240503 14:04:03.311666 14805 tablet_bootstrap.cc:654] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.312538 14808 tablet_bootstrap.cc:492] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0: Bootstrap starting.
I20240503 14:04:03.318183 14808 tablet_bootstrap.cc:654] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.328117 14805 tablet_bootstrap.cc:492] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0: No bootstrap required, opened a new log
I20240503 14:04:03.328655 14805 ts_tablet_manager.cc:1367] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.005s
I20240503 14:04:03.328943 14807 tablet_bootstrap.cc:492] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895: No bootstrap required, opened a new log
I20240503 14:04:03.329418 14807 ts_tablet_manager.cc:1367] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.012s
I20240503 14:04:03.331132 14805 raft_consensus.cc:348] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.331988 14805 raft_consensus.cc:374] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.331773 14807 raft_consensus.cc:348] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.332345 14808 tablet_bootstrap.cc:492] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0: No bootstrap required, opened a new log
I20240503 14:04:03.332312 14805 raft_consensus.cc:729] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.332541 14807 raft_consensus.cc:374] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.333067 14807 raft_consensus.cc:729] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7638e8e636a4601b99a740dd28fa895, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.333077 14808 ts_tablet_manager.cc:1367] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.000s
I20240503 14:04:03.333369 14805 consensus_queue.cc:260] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.333819 14807 consensus_queue.cc:260] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.335654 14805 ts_tablet_manager.cc:1398] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20240503 14:04:03.335749 14808 raft_consensus.cc:348] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.336942 14808 raft_consensus.cc:374] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.337213 14808 raft_consensus.cc:729] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b18bf626043b39784c11cebd55db0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.337852 14807 ts_tablet_manager.cc:1398] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240503 14:04:03.336670 14805 tablet_bootstrap.cc:492] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0: Bootstrap starting.
I20240503 14:04:03.337917 14808 consensus_queue.cc:260] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.338922 14807 tablet_bootstrap.cc:492] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895: Bootstrap starting.
I20240503 14:04:03.342840 14808 ts_tablet_manager.cc:1398] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240503 14:04:03.344074 14808 tablet_bootstrap.cc:492] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0: Bootstrap starting.
I20240503 14:04:03.344194 14805 tablet_bootstrap.cc:654] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.344341 14807 tablet_bootstrap.cc:654] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.350179 14808 tablet_bootstrap.cc:654] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.355582 14807 tablet_bootstrap.cc:492] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895: No bootstrap required, opened a new log
I20240503 14:04:03.356151 14807 ts_tablet_manager.cc:1367] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.004s
I20240503 14:04:03.358665 14807 raft_consensus.cc:348] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.359652 14807 raft_consensus.cc:374] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.359989 14807 raft_consensus.cc:729] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7638e8e636a4601b99a740dd28fa895, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.360883 14807 consensus_queue.cc:260] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.363930 14807 ts_tablet_manager.cc:1398] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:04:03.366137 14807 tablet_bootstrap.cc:492] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895: Bootstrap starting.
I20240503 14:04:03.367621 14808 tablet_bootstrap.cc:492] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0: No bootstrap required, opened a new log
I20240503 14:04:03.369223 14808 ts_tablet_manager.cc:1367] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.008s
I20240503 14:04:03.372097 14807 tablet_bootstrap.cc:654] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.372201 14808 raft_consensus.cc:348] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.373337 14808 raft_consensus.cc:374] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.373726 14808 raft_consensus.cc:729] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b18bf626043b39784c11cebd55db0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.374745 14808 consensus_queue.cc:260] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.382220 14805 tablet_bootstrap.cc:492] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0: No bootstrap required, opened a new log
I20240503 14:04:03.382694 14808 ts_tablet_manager.cc:1398] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240503 14:04:03.382901 14805 ts_tablet_manager.cc:1367] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0: Time spent bootstrapping tablet: real 0.046s user 0.021s sys 0.019s
I20240503 14:04:03.383711 14808 tablet_bootstrap.cc:492] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0: Bootstrap starting.
I20240503 14:04:03.383949 14807 tablet_bootstrap.cc:492] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895: No bootstrap required, opened a new log
I20240503 14:04:03.384472 14807 ts_tablet_manager.cc:1367] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.012s
I20240503 14:04:03.385507 14805 raft_consensus.cc:348] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.386354 14805 raft_consensus.cc:374] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.386758 14805 raft_consensus.cc:729] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.387143 14807 raft_consensus.cc:348] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.387697 14805 consensus_queue.cc:260] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.388180 14807 raft_consensus.cc:374] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.388622 14807 raft_consensus.cc:729] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7638e8e636a4601b99a740dd28fa895, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.389423 14807 consensus_queue.cc:260] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.391963 14808 tablet_bootstrap.cc:654] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.393962 14807 ts_tablet_manager.cc:1398] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 14:04:03.394893 14805 ts_tablet_manager.cc:1398] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240503 14:04:03.395079 14807 tablet_bootstrap.cc:492] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895: Bootstrap starting.
I20240503 14:04:03.395871 14805 tablet_bootstrap.cc:492] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0: Bootstrap starting.
I20240503 14:04:03.401119 14807 tablet_bootstrap.cc:654] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.402770 14805 tablet_bootstrap.cc:654] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.404150 14808 tablet_bootstrap.cc:492] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0: No bootstrap required, opened a new log
I20240503 14:04:03.404858 14808 ts_tablet_manager.cc:1367] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20240503 14:04:03.407416 14808 raft_consensus.cc:348] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.408350 14808 raft_consensus.cc:374] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.408700 14808 raft_consensus.cc:729] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b18bf626043b39784c11cebd55db0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.409505 14808 consensus_queue.cc:260] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.414520 14808 ts_tablet_manager.cc:1398] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20240503 14:04:03.416220 14808 tablet_bootstrap.cc:492] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0: Bootstrap starting.
I20240503 14:04:03.416671 14805 tablet_bootstrap.cc:492] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0: No bootstrap required, opened a new log
I20240503 14:04:03.417336 14805 ts_tablet_manager.cc:1367] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.009s
I20240503 14:04:03.420650 14805 raft_consensus.cc:348] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.421592 14805 raft_consensus.cc:374] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.421938 14805 raft_consensus.cc:729] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.422066 14808 tablet_bootstrap.cc:654] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.422745 14805 consensus_queue.cc:260] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.425830 14805 ts_tablet_manager.cc:1398] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240503 14:04:03.426949 14805 tablet_bootstrap.cc:492] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0: Bootstrap starting.
I20240503 14:04:03.428215 14807 tablet_bootstrap.cc:492] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895: No bootstrap required, opened a new log
I20240503 14:04:03.428727 14807 ts_tablet_manager.cc:1367] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.008s
I20240503 14:04:03.431533 14807 raft_consensus.cc:348] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.432322 14807 raft_consensus.cc:374] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.432626 14807 raft_consensus.cc:729] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7638e8e636a4601b99a740dd28fa895, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.433182 14805 tablet_bootstrap.cc:654] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.433315 14807 consensus_queue.cc:260] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.437155 14807 ts_tablet_manager.cc:1398] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20240503 14:04:03.438130 14807 tablet_bootstrap.cc:492] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895: Bootstrap starting.
I20240503 14:04:03.439757 14808 tablet_bootstrap.cc:492] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0: No bootstrap required, opened a new log
I20240503 14:04:03.440343 14808 ts_tablet_manager.cc:1367] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.012s
I20240503 14:04:03.443742 14808 raft_consensus.cc:348] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.444618 14808 raft_consensus.cc:374] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.444981 14808 raft_consensus.cc:729] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b18bf626043b39784c11cebd55db0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.445722 14808 consensus_queue.cc:260] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.448117 14805 tablet_bootstrap.cc:492] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0: No bootstrap required, opened a new log
I20240503 14:04:03.444991 14807 tablet_bootstrap.cc:654] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.448642 14805 ts_tablet_manager.cc:1367] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.012s
I20240503 14:04:03.452083 14808 ts_tablet_manager.cc:1398] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20240503 14:04:03.451860 14805 raft_consensus.cc:348] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } }
I20240503 14:04:03.452808 14805 raft_consensus.cc:374] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.453169 14808 tablet_bootstrap.cc:492] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0: Bootstrap starting.
I20240503 14:04:03.453236 14805 raft_consensus.cc:729] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.455092 14805 consensus_queue.cc:260] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } }
I20240503 14:04:03.457476 14805 ts_tablet_manager.cc:1398] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20240503 14:04:03.458287 14807 tablet_bootstrap.cc:492] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895: No bootstrap required, opened a new log
I20240503 14:04:03.458838 14807 ts_tablet_manager.cc:1367] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.007s
I20240503 14:04:03.458796 14814 raft_consensus.cc:482] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:03.459736 14805 tablet_bootstrap.cc:492] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0: Bootstrap starting.
I20240503 14:04:03.459419 14814 raft_consensus.cc:504] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.460170 14808 tablet_bootstrap.cc:654] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.465654 14814 leader_election.cc:288] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), 36443f8bb0a844a7947756620254c2e0 (127.13.158.67:45667)
I20240503 14:04:03.469800 14807 raft_consensus.cc:348] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.470793 14807 raft_consensus.cc:374] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.471132 14807 raft_consensus.cc:729] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7638e8e636a4601b99a740dd28fa895, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.471822 14805 tablet_bootstrap.cc:654] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.472029 14807 consensus_queue.cc:260] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.477236 14807 ts_tablet_manager.cc:1398] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240503 14:04:03.478351 14807 tablet_bootstrap.cc:492] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895: Bootstrap starting.
I20240503 14:04:03.486658 14808 tablet_bootstrap.cc:492] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0: No bootstrap required, opened a new log
I20240503 14:04:03.487299 14808 ts_tablet_manager.cc:1367] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.012s
I20240503 14:04:03.492743 14809 raft_consensus.cc:482] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:03.493664 14807 tablet_bootstrap.cc:654] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.493280 14809 raft_consensus.cc:504] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.496846 14809 leader_election.cc:288] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.499626 14611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2302ac8c2e544c68cf11427f3bf9143" candidate_uuid: "a7638e8e636a4601b99a740dd28fa895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0" is_pre_election: true
I20240503 14:04:03.505647 14809 raft_consensus.cc:482] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:03.504084 14808 raft_consensus.cc:348] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.514796 14808 raft_consensus.cc:374] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.518471 14808 raft_consensus.cc:729] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b18bf626043b39784c11cebd55db0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.514730 14809 raft_consensus.cc:504] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.521616 14814 raft_consensus.cc:482] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:03.522089 14814 raft_consensus.cc:504] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.522487 14805 tablet_bootstrap.cc:492] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0: No bootstrap required, opened a new log
I20240503 14:04:03.523111 14809 leader_election.cc:288] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.524152 14805 ts_tablet_manager.cc:1367] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.026s
I20240503 14:04:03.526787 14805 raft_consensus.cc:348] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.528261 14805 raft_consensus.cc:374] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.528504 14805 raft_consensus.cc:729] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.519567 14808 consensus_queue.cc:260] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.530128 14814 leader_election.cc:288] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), 36443f8bb0a844a7947756620254c2e0 (127.13.158.67:45667)
I20240503 14:04:03.530673 14611 raft_consensus.cc:2454] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7638e8e636a4601b99a740dd28fa895 in term 0.
I20240503 14:04:03.531147 14610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f1c0f5f2354dacbd45ac12c93e70bc" candidate_uuid: "a7638e8e636a4601b99a740dd28fa895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0" is_pre_election: true
I20240503 14:04:03.531808 14610 raft_consensus.cc:2454] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7638e8e636a4601b99a740dd28fa895 in term 0.
I20240503 14:04:03.533380 14649 leader_election.cc:302] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b18bf626043b39784c11cebd55db0, a7638e8e636a4601b99a740dd28fa895; no voters:
I20240503 14:04:03.534068 14805 consensus_queue.cc:260] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.537021 14805 ts_tablet_manager.cc:1398] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240503 14:04:03.538022 14805 tablet_bootstrap.cc:492] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0: Bootstrap starting.
I20240503 14:04:03.538384 14760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2302ac8c2e544c68cf11427f3bf9143" candidate_uuid: "a7638e8e636a4601b99a740dd28fa895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36443f8bb0a844a7947756620254c2e0" is_pre_election: true
I20240503 14:04:03.540673 14760 raft_consensus.cc:2454] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7638e8e636a4601b99a740dd28fa895 in term 0.
I20240503 14:04:03.547323 14834 raft_consensus.cc:482] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:03.548262 14834 raft_consensus.cc:504] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.545186 14651 leader_election.cc:302] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36443f8bb0a844a7947756620254c2e0, a7638e8e636a4601b99a740dd28fa895; no voters:
I20240503 14:04:03.545913 14759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f1c0f5f2354dacbd45ac12c93e70bc" candidate_uuid: "a7638e8e636a4601b99a740dd28fa895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36443f8bb0a844a7947756620254c2e0" is_pre_election: true
I20240503 14:04:03.543134 14814 raft_consensus.cc:2781] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:03.551718 14814 raft_consensus.cc:482] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:03.553190 14814 raft_consensus.cc:3037] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.554826 14611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7759c34f658a42c5a42e9f8ca831fe07" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0" is_pre_election: true
I20240503 14:04:03.554975 14610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2302ac8c2e544c68cf11427f3bf9143" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0" is_pre_election: true
I20240503 14:04:03.555377 14834 leader_election.cc:288] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.555713 14610 raft_consensus.cc:2454] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 0.
I20240503 14:04:03.556864 14611 raft_consensus.cc:2454] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 0.
I20240503 14:04:03.557355 14808 ts_tablet_manager.cc:1398] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0: Time spent starting tablet: real 0.070s user 0.007s sys 0.003s
I20240503 14:04:03.557739 14610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f023e4f1e70c47caad92f740e24b9acc" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0" is_pre_election: true
I20240503 14:04:03.564121 14808 tablet_bootstrap.cc:492] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0: Bootstrap starting.
I20240503 14:04:03.565948 14610 raft_consensus.cc:2454] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 0.
I20240503 14:04:03.563890 14805 tablet_bootstrap.cc:654] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0: Neither blocks nor log segments found. Creating new log.
W20240503 14:04:03.567047 14651 leader_election.cc:341] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 36443f8bb0a844a7947756620254c2e0 (127.13.158.67:45667): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:03.567746 14836 raft_consensus.cc:2781] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:03.567040 14723 leader_election.cc:302] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b18bf626043b39784c11cebd55db0, 36443f8bb0a844a7947756620254c2e0; no voters:
I20240503 14:04:03.571404 14807 tablet_bootstrap.cc:492] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895: No bootstrap required, opened a new log
I20240503 14:04:03.571449 14834 raft_consensus.cc:482] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:03.571971 14807 ts_tablet_manager.cc:1367] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895: Time spent bootstrapping tablet: real 0.094s user 0.028s sys 0.004s
I20240503 14:04:03.572522 14809 raft_consensus.cc:2781] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:03.572742 14723 leader_election.cc:302] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b18bf626043b39784c11cebd55db0, 36443f8bb0a844a7947756620254c2e0; no voters:
I20240503 14:04:03.572949 14809 raft_consensus.cc:482] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:03.573871 14686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7759c34f658a42c5a42e9f8ca831fe07" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895" is_pre_election: true
I20240503 14:04:03.572280 14834 raft_consensus.cc:504] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.573396 14809 raft_consensus.cc:3037] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.576323 14836 raft_consensus.cc:482] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:03.581707 14723 leader_election.cc:302] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b18bf626043b39784c11cebd55db0, 36443f8bb0a844a7947756620254c2e0; no voters:
I20240503 14:04:03.582538 14834 leader_election.cc:288] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.583063 14834 raft_consensus.cc:2781] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:03.583397 14834 raft_consensus.cc:482] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:03.583701 14834 raft_consensus.cc:3037] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.580746 14836 raft_consensus.cc:3037] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:04:03.585975 14724 leader_election.cc:341] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:03.587805 14686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfef82a71c8452f99024d7e7edc988f" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895" is_pre_election: true
I20240503 14:04:03.587599 14807 raft_consensus.cc:348] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.588691 14807 raft_consensus.cc:374] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.588752 14686 raft_consensus.cc:2454] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 0.
I20240503 14:04:03.589097 14807 raft_consensus.cc:729] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7638e8e636a4601b99a740dd28fa895, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.589862 14807 consensus_queue.cc:260] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.579176 14685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2302ac8c2e544c68cf11427f3bf9143" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895" is_pre_election: true
I20240503 14:04:03.590529 14808 tablet_bootstrap.cc:654] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.581779 14684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f023e4f1e70c47caad92f740e24b9acc" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895" is_pre_election: true
I20240503 14:04:03.592386 14814 raft_consensus.cc:504] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.594276 14724 leader_election.cc:302] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36443f8bb0a844a7947756620254c2e0, a7638e8e636a4601b99a740dd28fa895; no voters:
I20240503 14:04:03.596943 14684 raft_consensus.cc:2454] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 0.
W20240503 14:04:03.608170 14557 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:04:03.608783 14807 ts_tablet_manager.cc:1398] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895: Time spent starting tablet: real 0.036s user 0.011s sys 0.006s
I20240503 14:04:03.614358 14805 tablet_bootstrap.cc:492] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0: No bootstrap required, opened a new log
I20240503 14:04:03.616874 14814 leader_election.cc:288] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 election: Requested vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), 36443f8bb0a844a7947756620254c2e0 (127.13.158.67:45667)
I20240503 14:04:03.615990 14610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfef82a71c8452f99024d7e7edc988f" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0" is_pre_election: true
I20240503 14:04:03.619611 14562 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:03.625516 14807 tablet_bootstrap.cc:492] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895: Bootstrap starting.
I20240503 14:04:03.625615 14839 raft_consensus.cc:482] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:03.626514 14562 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:03.618012 14611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f1c0f5f2354dacbd45ac12c93e70bc" candidate_uuid: "a7638e8e636a4601b99a740dd28fa895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0"
I20240503 14:04:03.627238 14611 raft_consensus.cc:3037] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.627923 14838 raft_consensus.cc:2781] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:03.628249 14838 raft_consensus.cc:482] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:03.626210 14839 raft_consensus.cc:504] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.628544 14838 raft_consensus.cc:3037] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:04:03.630962 14723 leader_election.cc:341] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:03.631512 14759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f1c0f5f2354dacbd45ac12c93e70bc" candidate_uuid: "a7638e8e636a4601b99a740dd28fa895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36443f8bb0a844a7947756620254c2e0"
I20240503 14:04:03.629504 14805 ts_tablet_manager.cc:1367] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0: Time spent bootstrapping tablet: real 0.091s user 0.012s sys 0.005s
W20240503 14:04:03.632932 14651 leader_election.cc:341] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 36443f8bb0a844a7947756620254c2e0 (127.13.158.67:45667): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:03.635197 14805 raft_consensus.cc:348] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.636000 14805 raft_consensus.cc:374] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.636265 14805 raft_consensus.cc:729] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.637287 14839 leader_election.cc:288] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.636943 14805 consensus_queue.cc:260] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.638003 14610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3fa6230957409db8eff9dccbf62b32" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0" is_pre_election: true
I20240503 14:04:03.638687 14610 raft_consensus.cc:2454] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 0.
I20240503 14:04:03.630571 14842 raft_consensus.cc:2781] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:03.639359 14805 ts_tablet_manager.cc:1398] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 14:04:03.639282 14842 raft_consensus.cc:482] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:03.639941 14723 leader_election.cc:302] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b18bf626043b39784c11cebd55db0, 36443f8bb0a844a7947756620254c2e0; no voters:
I20240503 14:04:03.640193 14842 raft_consensus.cc:3037] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.641583 14843 raft_consensus.cc:2781] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:03.641896 14843 raft_consensus.cc:482] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:03.641943 14684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3fa6230957409db8eff9dccbf62b32" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895" is_pre_election: true
I20240503 14:04:03.642501 14843 raft_consensus.cc:3037] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.642705 14684 raft_consensus.cc:2454] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 0.
I20240503 14:04:03.646826 14836 raft_consensus.cc:504] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.648049 14685 raft_consensus.cc:2379] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 36443f8bb0a844a7947756620254c2e0 in current term 1: Already voted for candidate a7638e8e636a4601b99a740dd28fa895 in this term.
I20240503 14:04:03.651710 14807 tablet_bootstrap.cc:654] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.651546 14809 raft_consensus.cc:504] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.651686 14834 raft_consensus.cc:504] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.652523 14808 tablet_bootstrap.cc:492] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0: No bootstrap required, opened a new log
I20240503 14:04:03.654796 14610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2302ac8c2e544c68cf11427f3bf9143" candidate_uuid: "a7638e8e636a4601b99a740dd28fa895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0"
I20240503 14:04:03.655536 14610 raft_consensus.cc:3037] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.658447 14808 ts_tablet_manager.cc:1367] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0: Time spent bootstrapping tablet: real 0.099s user 0.012s sys 0.015s
I20240503 14:04:03.659739 14836 leader_election.cc:288] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 election: Requested vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), 36443f8bb0a844a7947756620254c2e0 (127.13.158.67:45667)
I20240503 14:04:03.660084 14834 leader_election.cc:288] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 election: Requested vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.662256 14685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7759c34f658a42c5a42e9f8ca831fe07" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895"
I20240503 14:04:03.662966 14685 raft_consensus.cc:3037] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.662487 14808 raft_consensus.cc:348] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.665030 14808 raft_consensus.cc:374] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.665695 14808 raft_consensus.cc:729] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b18bf626043b39784c11cebd55db0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.668165 14759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2302ac8c2e544c68cf11427f3bf9143" candidate_uuid: "a7638e8e636a4601b99a740dd28fa895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36443f8bb0a844a7947756620254c2e0"
I20240503 14:04:03.665529 14609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7759c34f658a42c5a42e9f8ca831fe07" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0"
I20240503 14:04:03.667446 14684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2302ac8c2e544c68cf11427f3bf9143" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895"
I20240503 14:04:03.670233 14759 raft_consensus.cc:2379] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a7638e8e636a4601b99a740dd28fa895 in current term 1: Already voted for candidate 36443f8bb0a844a7947756620254c2e0 in this term.
I20240503 14:04:03.664852 14809 leader_election.cc:288] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 election: Requested vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.668028 14608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2302ac8c2e544c68cf11427f3bf9143" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0"
I20240503 14:04:03.671983 14684 raft_consensus.cc:2379] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 36443f8bb0a844a7947756620254c2e0 in current term 1: Already voted for candidate a7638e8e636a4601b99a740dd28fa895 in this term.
I20240503 14:04:03.671005 14609 raft_consensus.cc:3037] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.672784 14808 consensus_queue.cc:260] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.677104 14723 leader_election.cc:302] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36443f8bb0a844a7947756620254c2e0; no voters: 186b18bf626043b39784c11cebd55db0, a7638e8e636a4601b99a740dd28fa895
I20240503 14:04:03.680374 14808 ts_tablet_manager.cc:1398] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20240503 14:04:03.681222 14807 tablet_bootstrap.cc:492] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895: No bootstrap required, opened a new log
I20240503 14:04:03.681836 14809 raft_consensus.cc:2726] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:04:03.681577 14808 tablet_bootstrap.cc:492] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0: Bootstrap starting.
I20240503 14:04:03.685546 14807 ts_tablet_manager.cc:1367] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895: Time spent bootstrapping tablet: real 0.062s user 0.025s sys 0.002s
I20240503 14:04:03.689227 14807 raft_consensus.cc:348] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } }
I20240503 14:04:03.690814 14807 raft_consensus.cc:374] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.691020 14807 raft_consensus.cc:729] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7638e8e636a4601b99a740dd28fa895, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.691519 14807 consensus_queue.cc:260] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } }
I20240503 14:04:03.696502 14807 ts_tablet_manager.cc:1398] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895: Time spent starting tablet: real 0.011s user 0.001s sys 0.004s
I20240503 14:04:03.697186 14808 tablet_bootstrap.cc:654] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:03.705314 14808 tablet_bootstrap.cc:492] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0: No bootstrap required, opened a new log
I20240503 14:04:03.705729 14808 ts_tablet_manager.cc:1367] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.002s
I20240503 14:04:03.706820 14685 raft_consensus.cc:2454] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 1.
I20240503 14:04:03.706367 14838 raft_consensus.cc:504] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.707922 14609 raft_consensus.cc:2454] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 1.
I20240503 14:04:03.708719 14808 raft_consensus.cc:348] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } }
I20240503 14:04:03.706470 14842 raft_consensus.cc:504] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.709527 14808 raft_consensus.cc:374] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:03.709837 14808 raft_consensus.cc:729] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b18bf626043b39784c11cebd55db0, State: Initialized, Role: FOLLOWER
I20240503 14:04:03.709970 14684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f023e4f1e70c47caad92f740e24b9acc" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895"
I20240503 14:04:03.710966 14842 leader_election.cc:288] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 election: Requested vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.711225 14684 raft_consensus.cc:3037] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.708899 14611 raft_consensus.cc:2454] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7638e8e636a4601b99a740dd28fa895 in term 1.
I20240503 14:04:03.711647 14723 leader_election.cc:302] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b18bf626043b39784c11cebd55db0, 36443f8bb0a844a7947756620254c2e0; no voters:
I20240503 14:04:03.712126 14608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfef82a71c8452f99024d7e7edc988f" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0"
I20240503 14:04:03.712725 14842 raft_consensus.cc:2781] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:03.713346 14608 raft_consensus.cc:3037] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.713369 14609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f023e4f1e70c47caad92f740e24b9acc" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0"
I20240503 14:04:03.713661 14842 raft_consensus.cc:686] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Becoming Leader. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Running, Role: LEADER
I20240503 14:04:03.714084 14609 raft_consensus.cc:3037] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.710611 14808 consensus_queue.cc:260] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } }
I20240503 14:04:03.714517 14842 consensus_queue.cc:237] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.715289 14836 raft_consensus.cc:482] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:03.715725 14836 raft_consensus.cc:504] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } }
I20240503 14:04:03.716573 14808 ts_tablet_manager.cc:1398] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20240503 14:04:03.717464 14809 raft_consensus.cc:482] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:03.717940 14809 raft_consensus.cc:504] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.719197 14759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a8632a21724f14ab7ddd5dcee692df" candidate_uuid: "a7638e8e636a4601b99a740dd28fa895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36443f8bb0a844a7947756620254c2e0" is_pre_election: true
I20240503 14:04:03.719836 14809 leader_election.cc:288] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.719904 14759 raft_consensus.cc:2454] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7638e8e636a4601b99a740dd28fa895 in term 0.
I20240503 14:04:03.722209 14651 leader_election.cc:302] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36443f8bb0a844a7947756620254c2e0, a7638e8e636a4601b99a740dd28fa895; no voters:
I20240503 14:04:03.722488 14611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17937dd0f02348718b2704f4cddb0397" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0" is_pre_election: true
I20240503 14:04:03.723165 14611 raft_consensus.cc:2454] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 0.
I20240503 14:04:03.723790 14840 raft_consensus.cc:2781] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:03.724220 14840 raft_consensus.cc:482] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:03.724587 14723 leader_election.cc:302] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b18bf626043b39784c11cebd55db0, 36443f8bb0a844a7947756620254c2e0; no voters:
I20240503 14:04:03.725499 14809 raft_consensus.cc:2781] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:03.725854 14809 raft_consensus.cc:482] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:03.726176 14809 raft_consensus.cc:3037] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.715095 14649 leader_election.cc:302] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b18bf626043b39784c11cebd55db0, a7638e8e636a4601b99a740dd28fa895; no voters: 36443f8bb0a844a7947756620254c2e0
I20240503 14:04:03.727321 14814 raft_consensus.cc:2781] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:03.729635 14611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a8632a21724f14ab7ddd5dcee692df" candidate_uuid: "a7638e8e636a4601b99a740dd28fa895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0" is_pre_election: true
I20240503 14:04:03.717598 14836 leader_election.cc:288] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36443f8bb0a844a7947756620254c2e0 (127.13.158.67:45667), 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101)
I20240503 14:04:03.708664 14838 leader_election.cc:288] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 election: Requested vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.732607 14611 raft_consensus.cc:2454] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7638e8e636a4601b99a740dd28fa895 in term 0.
I20240503 14:04:03.724543 14840 raft_consensus.cc:3037] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.735381 14843 raft_consensus.cc:504] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.738516 14843 leader_election.cc:288] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 election: Requested vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.738777 14610 raft_consensus.cc:2454] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7638e8e636a4601b99a740dd28fa895 in term 1.
I20240503 14:04:03.739467 14611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3fa6230957409db8eff9dccbf62b32" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0"
I20240503 14:04:03.740165 14611 raft_consensus.cc:3037] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.740268 14649 leader_election.cc:302] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 186b18bf626043b39784c11cebd55db0, a7638e8e636a4601b99a740dd28fa895; no voters: 36443f8bb0a844a7947756620254c2e0
I20240503 14:04:03.741569 14836 raft_consensus.cc:2781] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:03.742048 14836 raft_consensus.cc:686] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 1 LEADER]: Becoming Leader. State: Replica: a7638e8e636a4601b99a740dd28fa895, State: Running, Role: LEADER
I20240503 14:04:03.741981 14508 catalog_manager.cc:5484] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36443f8bb0a844a7947756620254c2e0 (127.13.158.67). New cstate: current_term: 1 leader_uuid: "36443f8bb0a844a7947756620254c2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:03.742918 14836 consensus_queue.cc:237] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.749353 14685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfef82a71c8452f99024d7e7edc988f" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895"
I20240503 14:04:03.750072 14685 raft_consensus.cc:3037] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.754689 14686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17937dd0f02348718b2704f4cddb0397" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895" is_pre_election: true
I20240503 14:04:03.755120 14683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3fa6230957409db8eff9dccbf62b32" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895"
I20240503 14:04:03.755824 14683 raft_consensus.cc:3037] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.755573 14686 raft_consensus.cc:2454] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 0.
I20240503 14:04:03.761839 14814 raft_consensus.cc:686] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 1 LEADER]: Becoming Leader. State: Replica: a7638e8e636a4601b99a740dd28fa895, State: Running, Role: LEADER
I20240503 14:04:03.763021 14814 consensus_queue.cc:237] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.773499 14509 catalog_manager.cc:5484] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7638e8e636a4601b99a740dd28fa895 (127.13.158.66). New cstate: current_term: 1 leader_uuid: "a7638e8e636a4601b99a740dd28fa895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:03.789572 14509 catalog_manager.cc:5484] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7638e8e636a4601b99a740dd28fa895 (127.13.158.66). New cstate: current_term: 1 leader_uuid: "a7638e8e636a4601b99a740dd28fa895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:03.833360 14608 raft_consensus.cc:2454] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 1.
I20240503 14:04:03.833163 14809 raft_consensus.cc:504] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.833786 14683 raft_consensus.cc:2454] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 1.
I20240503 14:04:03.834887 14684 raft_consensus.cc:2454] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 1.
I20240503 14:04:03.835539 14809 leader_election.cc:288] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 election: Requested vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.835958 14840 raft_consensus.cc:504] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } }
I20240503 14:04:03.836578 14724 leader_election.cc:302] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36443f8bb0a844a7947756620254c2e0, a7638e8e636a4601b99a740dd28fa895; no voters:
I20240503 14:04:03.833089 14685 raft_consensus.cc:2454] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 1.
I20240503 14:04:03.837536 14724 leader_election.cc:302] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36443f8bb0a844a7947756620254c2e0, a7638e8e636a4601b99a740dd28fa895; no voters:
I20240503 14:04:03.838030 14840 leader_election.cc:288] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 election: Requested vote from peers 36443f8bb0a844a7947756620254c2e0 (127.13.158.67:45667), 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101)
I20240503 14:04:03.838500 14724 leader_election.cc:302] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36443f8bb0a844a7947756620254c2e0, a7638e8e636a4601b99a740dd28fa895; no voters:
I20240503 14:04:03.838500 14838 raft_consensus.cc:2781] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:03.839401 14842 raft_consensus.cc:482] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:03.835853 14609 raft_consensus.cc:2454] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 1.
I20240503 14:04:03.839639 14838 raft_consensus.cc:686] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Becoming Leader. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Running, Role: LEADER
I20240503 14:04:03.840087 14842 raft_consensus.cc:504] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.840063 14608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a8632a21724f14ab7ddd5dcee692df" candidate_uuid: "a7638e8e636a4601b99a740dd28fa895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0"
I20240503 14:04:03.835470 14611 raft_consensus.cc:2454] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 1.
I20240503 14:04:03.838693 14843 raft_consensus.cc:2781] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:03.840504 14834 raft_consensus.cc:2781] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:03.841178 14759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a8632a21724f14ab7ddd5dcee692df" candidate_uuid: "a7638e8e636a4601b99a740dd28fa895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36443f8bb0a844a7947756620254c2e0"
I20240503 14:04:03.842027 14834 raft_consensus.cc:686] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Becoming Leader. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Running, Role: LEADER
I20240503 14:04:03.841328 14608 raft_consensus.cc:3037] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.842271 14759 raft_consensus.cc:3037] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.842394 14842 leader_election.cc:288] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101), a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:03.842079 14838 consensus_queue.cc:237] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.841648 14843 raft_consensus.cc:686] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Becoming Leader. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Running, Role: LEADER
I20240503 14:04:03.842904 14834 consensus_queue.cc:237] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.844954 14843 consensus_queue.cc:237] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.839176 14684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17937dd0f02348718b2704f4cddb0397" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895"
I20240503 14:04:03.848073 14609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f1c0f5f2354dacbd45ac12c93e70bc" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0" is_pre_election: true
I20240503 14:04:03.848373 14684 raft_consensus.cc:3037] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.848894 14609 raft_consensus.cc:2379] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 36443f8bb0a844a7947756620254c2e0 in current term 1: Already voted for candidate a7638e8e636a4601b99a740dd28fa895 in this term.
I20240503 14:04:03.849828 14611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17937dd0f02348718b2704f4cddb0397" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b18bf626043b39784c11cebd55db0"
I20240503 14:04:03.850406 14611 raft_consensus.cc:3037] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.852715 14685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f1c0f5f2354dacbd45ac12c93e70bc" candidate_uuid: "36443f8bb0a844a7947756620254c2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7638e8e636a4601b99a740dd28fa895" is_pre_election: true
I20240503 14:04:03.855190 14723 leader_election.cc:302] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36443f8bb0a844a7947756620254c2e0; no voters: 186b18bf626043b39784c11cebd55db0, a7638e8e636a4601b99a740dd28fa895
I20240503 14:04:03.857648 14834 raft_consensus.cc:3037] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:03.864159 14509 catalog_manager.cc:5484] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36443f8bb0a844a7947756620254c2e0 (127.13.158.67). New cstate: current_term: 1 leader_uuid: "36443f8bb0a844a7947756620254c2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:03.877707 14509 catalog_manager.cc:5484] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36443f8bb0a844a7947756620254c2e0 (127.13.158.67). New cstate: current_term: 1 leader_uuid: "36443f8bb0a844a7947756620254c2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:03.879266 14509 catalog_manager.cc:5484] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36443f8bb0a844a7947756620254c2e0 (127.13.158.67). New cstate: current_term: 1 leader_uuid: "36443f8bb0a844a7947756620254c2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:03.934038 14608 raft_consensus.cc:2454] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7638e8e636a4601b99a740dd28fa895 in term 1.
I20240503 14:04:03.934038 14684 raft_consensus.cc:2454] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 1.
I20240503 14:04:03.934038 14759 raft_consensus.cc:2454] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7638e8e636a4601b99a740dd28fa895 in term 1.
I20240503 14:04:03.937423 14651 leader_election.cc:302] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36443f8bb0a844a7947756620254c2e0, a7638e8e636a4601b99a740dd28fa895; no voters:
I20240503 14:04:03.938361 14840 raft_consensus.cc:2781] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:03.938786 14724 leader_election.cc:302] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36443f8bb0a844a7947756620254c2e0, a7638e8e636a4601b99a740dd28fa895; no voters:
I20240503 14:04:03.938843 14840 raft_consensus.cc:686] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 1 LEADER]: Becoming Leader. State: Replica: a7638e8e636a4601b99a740dd28fa895, State: Running, Role: LEADER
I20240503 14:04:03.939783 14838 raft_consensus.cc:2781] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:03.940280 14838 raft_consensus.cc:686] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Becoming Leader. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Running, Role: LEADER
I20240503 14:04:03.940044 14840 consensus_queue.cc:237] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } }
I20240503 14:04:03.941135 14838 consensus_queue.cc:237] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:03.948395 14509 catalog_manager.cc:5484] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7638e8e636a4601b99a740dd28fa895 (127.13.158.66). New cstate: current_term: 1 leader_uuid: "a7638e8e636a4601b99a740dd28fa895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:03.949119 14508 catalog_manager.cc:5484] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36443f8bb0a844a7947756620254c2e0 (127.13.158.67). New cstate: current_term: 1 leader_uuid: "36443f8bb0a844a7947756620254c2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:03.984946 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:03.991561 14859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:03.992444 14860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:03.993383 14862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:03.994313 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:03.995213 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:03.995414 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:03.995592 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745043995578 us; error 0 us; skew 500 ppm
I20240503 14:04:03.996162 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:03.998876 13945 webserver.cc:460] Webserver started at http://127.13.158.68:42051/ using document root <none> and password file <none>
I20240503 14:04:03.999361 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:03.999552 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:03.999840 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:04.000969 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-3-root/instance:
uuid: "1ef24c13b6b74446831df6680917bc9f"
format_stamp: "Formatted at 2024-05-03 14:04:03 on dist-test-slave-jm3r"
I20240503 14:04:04.005658 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20240503 14:04:04.009025 14867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:04.010005 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 14:04:04.010334 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-3-root
uuid: "1ef24c13b6b74446831df6680917bc9f"
format_stamp: "Formatted at 2024-05-03 14:04:03 on dist-test-slave-jm3r"
I20240503 14:04:04.010859 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:04.028347 14834 raft_consensus.cc:2726] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:04.028321 14611 raft_consensus.cc:2454] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36443f8bb0a844a7947756620254c2e0 in term 1.
I20240503 14:04:04.045722 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:04.047072 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:04.048283 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:04.051093 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:04.051306 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:04.051584 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:04.051754 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:04.102560 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.68:37341
I20240503 14:04:04.102624 14929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.68:37341 every 8 connection(s)
I20240503 14:04:04.118489 14930 heartbeater.cc:347] Connected to a master server at 127.13.158.126:39965
I20240503 14:04:04.118921 14930 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:04.119705 14930 heartbeater.cc:511] Master 127.13.158.126:39965 requested a full tablet report, sending...
I20240503 14:04:04.121909 14508 ts_manager.cc:194] Registered new tserver with Master: 1ef24c13b6b74446831df6680917bc9f (127.13.158.68:37341)
I20240503 14:04:04.123754 14508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44592
I20240503 14:04:04.123661 14834 consensus_queue.cc:1035] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:04.143010 14843 consensus_queue.cc:1035] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:04.244294 14843 consensus_queue.cc:1035] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:04.250856 14843 consensus_queue.cc:1035] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:04.258486 14840 consensus_queue.cc:1035] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:04.275632 14843 consensus_queue.cc:1035] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:04.276575 14840 consensus_queue.cc:1035] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:04.315697 14839 consensus_queue.cc:1035] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:04.326066 14814 consensus_queue.cc:1035] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:04.334604 14814 consensus_queue.cc:1035] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:04.342383 14814 consensus_queue.cc:1035] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:04.354842 14840 consensus_queue.cc:1035] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:04.356564 14839 consensus_queue.cc:1035] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:04.372691 14839 consensus_queue.cc:1035] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:04.508746 14839 consensus_queue.cc:1035] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:04.520536 14834 consensus_queue.cc:1035] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:05.127889 14930 heartbeater.cc:503] Master 127.13.158.126:39965 was elected leader, sending a full tablet report...
I20240503 14:04:05.627568 14562 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:05.645946 14760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17937dd0f02348718b2704f4cddb0397"
dest_uuid: "36443f8bb0a844a7947756620254c2e0"
mode: GRACEFUL
new_leader_uuid: "186b18bf626043b39784c11cebd55db0"
from {username='slave'} at 127.0.0.1:35848
I20240503 14:04:05.646651 14760 raft_consensus.cc:595] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Received request to transfer leadership to TS 186b18bf626043b39784c11cebd55db0
I20240503 14:04:05.649207 14760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f023e4f1e70c47caad92f740e24b9acc"
dest_uuid: "36443f8bb0a844a7947756620254c2e0"
mode: GRACEFUL
new_leader_uuid: "186b18bf626043b39784c11cebd55db0"
from {username='slave'} at 127.0.0.1:35848
I20240503 14:04:05.649727 14760 raft_consensus.cc:595] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Received request to transfer leadership to TS 186b18bf626043b39784c11cebd55db0
I20240503 14:04:05.650770 14562 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 14:04:05.651252 14562 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:05.761344 14843 raft_consensus.cc:979] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0: : Instructing follower 186b18bf626043b39784c11cebd55db0 to start an election
I20240503 14:04:05.761920 14843 raft_consensus.cc:1067] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Signalling peer 186b18bf626043b39784c11cebd55db0 to start an election
I20240503 14:04:05.763710 14608 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f023e4f1e70c47caad92f740e24b9acc"
dest_uuid: "186b18bf626043b39784c11cebd55db0"
from {username='slave'} at 127.0.0.1:34024
I20240503 14:04:05.764436 14608 raft_consensus.cc:482] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:05.764762 14608 raft_consensus.cc:3037] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:05.770776 14608 raft_consensus.cc:504] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:05.772737 14608 leader_election.cc:288] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [CANDIDATE]: Term 2 election: Requested vote from peers a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235), 36443f8bb0a844a7947756620254c2e0 (127.13.158.67:45667)
I20240503 14:04:05.787153 14760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f023e4f1e70c47caad92f740e24b9acc" candidate_uuid: "186b18bf626043b39784c11cebd55db0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36443f8bb0a844a7947756620254c2e0"
I20240503 14:04:05.787828 14760 raft_consensus.cc:3032] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:05.788102 14760 raft_consensus.cc:729] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Running, Role: LEADER
I20240503 14:04:05.788314 14685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f023e4f1e70c47caad92f740e24b9acc" candidate_uuid: "186b18bf626043b39784c11cebd55db0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a7638e8e636a4601b99a740dd28fa895"
I20240503 14:04:05.788904 14685 raft_consensus.cc:3037] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:05.788760 14760 consensus_queue.cc:260] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:05.789825 14760 raft_consensus.cc:3037] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:05.794996 14685 raft_consensus.cc:2454] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b18bf626043b39784c11cebd55db0 in term 2.
I20240503 14:04:05.794984 14760 raft_consensus.cc:2454] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b18bf626043b39784c11cebd55db0 in term 2.
I20240503 14:04:05.796396 14575 leader_election.cc:302] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b18bf626043b39784c11cebd55db0, a7638e8e636a4601b99a740dd28fa895; no voters:
I20240503 14:04:05.797935 14962 raft_consensus.cc:2781] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:04:05.799062 14962 raft_consensus.cc:686] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 2 LEADER]: Becoming Leader. State: Replica: 186b18bf626043b39784c11cebd55db0, State: Running, Role: LEADER
I20240503 14:04:05.799779 14962 consensus_queue.cc:237] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:05.806581 14506 catalog_manager.cc:5484] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 reported cstate change: term changed from 1 to 2, leader changed from 36443f8bb0a844a7947756620254c2e0 (127.13.158.67) to 186b18bf626043b39784c11cebd55db0 (127.13.158.65). New cstate: current_term: 2 leader_uuid: "186b18bf626043b39784c11cebd55db0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:05.889839 14958 raft_consensus.cc:979] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0: : Instructing follower 186b18bf626043b39784c11cebd55db0 to start an election
I20240503 14:04:05.890343 14843 raft_consensus.cc:1067] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Signalling peer 186b18bf626043b39784c11cebd55db0 to start an election
I20240503 14:04:05.892166 14608 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17937dd0f02348718b2704f4cddb0397"
dest_uuid: "186b18bf626043b39784c11cebd55db0"
from {username='slave'} at 127.0.0.1:34024
I20240503 14:04:05.892913 14608 raft_consensus.cc:482] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:05.893436 14608 raft_consensus.cc:3037] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:05.898165 14608 raft_consensus.cc:504] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:05.899861 14608 leader_election.cc:288] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [CANDIDATE]: Term 2 election: Requested vote from peers a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235), 36443f8bb0a844a7947756620254c2e0 (127.13.158.67:45667)
I20240503 14:04:05.900982 14685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17937dd0f02348718b2704f4cddb0397" candidate_uuid: "186b18bf626043b39784c11cebd55db0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a7638e8e636a4601b99a740dd28fa895"
I20240503 14:04:05.901114 14760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17937dd0f02348718b2704f4cddb0397" candidate_uuid: "186b18bf626043b39784c11cebd55db0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36443f8bb0a844a7947756620254c2e0"
I20240503 14:04:05.901538 14685 raft_consensus.cc:3037] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:05.901665 14760 raft_consensus.cc:3032] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:05.902118 14760 raft_consensus.cc:729] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36443f8bb0a844a7947756620254c2e0, State: Running, Role: LEADER
I20240503 14:04:05.902892 14760 consensus_queue.cc:260] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:05.904115 14760 raft_consensus.cc:3037] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:05.906491 14685 raft_consensus.cc:2454] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b18bf626043b39784c11cebd55db0 in term 2.
I20240503 14:04:05.907737 14575 leader_election.cc:302] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b18bf626043b39784c11cebd55db0, a7638e8e636a4601b99a740dd28fa895; no voters:
I20240503 14:04:05.908900 14962 raft_consensus.cc:2781] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:04:05.909132 14760 raft_consensus.cc:2454] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b18bf626043b39784c11cebd55db0 in term 2.
I20240503 14:04:05.909430 14962 raft_consensus.cc:686] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 2 LEADER]: Becoming Leader. State: Replica: 186b18bf626043b39784c11cebd55db0, State: Running, Role: LEADER
I20240503 14:04:05.910544 14962 consensus_queue.cc:237] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } }
I20240503 14:04:05.918411 14506 catalog_manager.cc:5484] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 reported cstate change: term changed from 1 to 2, leader changed from 36443f8bb0a844a7947756620254c2e0 (127.13.158.67) to 186b18bf626043b39784c11cebd55db0 (127.13.158.65). New cstate: current_term: 2 leader_uuid: "186b18bf626043b39784c11cebd55db0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:06.273747 14685 raft_consensus.cc:1261] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 2 FOLLOWER]: Refusing update from remote peer 186b18bf626043b39784c11cebd55db0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:06.275195 14962 consensus_queue.cc:1035] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:06.289366 14760 raft_consensus.cc:1261] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 2 FOLLOWER]: Refusing update from remote peer 186b18bf626043b39784c11cebd55db0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:06.291344 14972 consensus_queue.cc:1035] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:04:06.458819 14760 raft_consensus.cc:1261] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 2 FOLLOWER]: Refusing update from remote peer 186b18bf626043b39784c11cebd55db0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:06.460553 14972 consensus_queue.cc:1035] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:06.469724 14685 raft_consensus.cc:1261] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895 [term 2 FOLLOWER]: Refusing update from remote peer 186b18bf626043b39784c11cebd55db0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:06.471606 14963 consensus_queue.cc:1035] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:07.652335 14562 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:07.658207 14562 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:07.658730 14562 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:09.640108 14684 consensus_queue.cc:237] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } attrs { replace: true } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } }
I20240503 14:04:09.649545 14760 raft_consensus.cc:1261] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Refusing update from remote peer a7638e8e636a4601b99a740dd28fa895: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:04:09.650146 14608 raft_consensus.cc:1261] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Refusing update from remote peer a7638e8e636a4601b99a740dd28fa895: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:04:09.651209 14840 consensus_queue.cc:1035] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:04:09.652413 14983 consensus_queue.cc:1035] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:04:09.659844 14562 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240503 14:04:09.661764 14649 consensus_peers.cc:485] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 -> Peer 1ef24c13b6b74446831df6680917bc9f (127.13.158.68:37341): Couldn't send request to peer 1ef24c13b6b74446831df6680917bc9f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99f1c0f5f2354dacbd45ac12c93e70bc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:09.664458 14983 raft_consensus.cc:2932] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } attrs { replace: true } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.666779 14608 raft_consensus.cc:2932] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } attrs { replace: true } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.668649 14562 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:09.669268 14562 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:09.667941 14760 raft_consensus.cc:2932] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } attrs { replace: true } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.680231 14508 catalog_manager.cc:5484] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New cstate: current_term: 1 leader_uuid: "a7638e8e636a4601b99a740dd28fa895" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } attrs { replace: true } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.697355 14608 consensus_queue.cc:237] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } attrs { replace: true } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } }
I20240503 14:04:09.705783 14760 raft_consensus.cc:1261] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 2 FOLLOWER]: Refusing update from remote peer 186b18bf626043b39784c11cebd55db0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:04:09.707322 14995 consensus_queue.cc:1035] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:09.707723 14684 raft_consensus.cc:1261] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 2 FOLLOWER]: Refusing update from remote peer 186b18bf626043b39784c11cebd55db0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:04:09.709740 14972 consensus_queue.cc:1035] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:09.725392 14759 raft_consensus.cc:2932] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } attrs { replace: true } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
W20240503 14:04:09.729826 14574 consensus_peers.cc:485] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 -> Peer 1ef24c13b6b74446831df6680917bc9f (127.13.158.68:37341): Couldn't send request to peer 1ef24c13b6b74446831df6680917bc9f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f023e4f1e70c47caad92f740e24b9acc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:09.723882 14972 raft_consensus.cc:2932] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } attrs { replace: true } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.743005 14506 catalog_manager.cc:5484] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 reported cstate change: config changed from index -1 to 3, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New cstate: current_term: 2 leader_uuid: "186b18bf626043b39784c11cebd55db0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } attrs { replace: true } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.743577 14684 raft_consensus.cc:2932] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } attrs { replace: true } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.766366 14759 consensus_queue.cc:237] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } }
I20240503 14:04:09.774200 14684 raft_consensus.cc:1261] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Refusing update from remote peer 36443f8bb0a844a7947756620254c2e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:04:09.776177 14608 raft_consensus.cc:1261] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Refusing update from remote peer 36443f8bb0a844a7947756620254c2e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:04:09.776216 15003 consensus_queue.cc:1035] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:04:09.778394 14843 consensus_queue.cc:1035] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240503 14:04:09.784788 14723 consensus_peers.cc:485] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 -> Peer 1ef24c13b6b74446831df6680917bc9f (127.13.158.68:37341): Couldn't send request to peer 1ef24c13b6b74446831df6680917bc9f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b3fa6230957409db8eff9dccbf62b32. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:09.788144 15007 raft_consensus.cc:2932] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.789317 14684 raft_consensus.cc:2932] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.789651 14608 raft_consensus.cc:2932] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.803047 14509 catalog_manager.cc:5484] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New cstate: current_term: 1 leader_uuid: "36443f8bb0a844a7947756620254c2e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.805904 14684 consensus_queue.cc:237] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } attrs { replace: true } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } }
W20240503 14:04:09.814901 14649 consensus_peers.cc:485] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 -> Peer 1ef24c13b6b74446831df6680917bc9f (127.13.158.68:37341): Couldn't send request to peer 1ef24c13b6b74446831df6680917bc9f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2302ac8c2e544c68cf11427f3bf9143. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:09.817431 14759 raft_consensus.cc:1261] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Refusing update from remote peer a7638e8e636a4601b99a740dd28fa895: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:04:09.817373 14608 raft_consensus.cc:1261] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Refusing update from remote peer a7638e8e636a4601b99a740dd28fa895: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:04:09.818737 14840 consensus_queue.cc:1035] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:09.819753 14983 consensus_queue.cc:1035] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:09.828208 14840 raft_consensus.cc:2932] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } attrs { replace: true } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.830956 14608 raft_consensus.cc:2932] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } attrs { replace: true } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.831759 14759 raft_consensus.cc:2932] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } attrs { replace: true } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } } }
I20240503 14:04:09.854758 14506 catalog_manager.cc:5484] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ef24c13b6b74446831df6680917bc9f (127.13.158.68) added. New cstate: current_term: 1 leader_uuid: "a7638e8e636a4601b99a740dd28fa895" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:10.137945 13945 tablet_server.cc:178] TabletServer@127.13.158.65:0 shutting down...
W20240503 14:04:10.146014 14493 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20240503 14:04:10.149601 14557 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 14:04:10.150071 14557 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 14:04:10.219412 15012 ts_tablet_manager.cc:902] T 99f1c0f5f2354dacbd45ac12c93e70bc P 1ef24c13b6b74446831df6680917bc9f: Initiating tablet copy from peer a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:10.225405 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240503 14:04:10.220882 14649 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.13.158.65:44101, user_credentials={real_user=slave}} blocked reactor thread for 52118.8us
W20240503 14:04:10.222030 14723 consensus_peers.cc:485] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 -> Peer 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101): Couldn't send request to peer 186b18bf626043b39784c11cebd55db0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:10.222942 15015 ts_tablet_manager.cc:902] T f023e4f1e70c47caad92f740e24b9acc P 1ef24c13b6b74446831df6680917bc9f: Initiating tablet copy from peer 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101)
I20240503 14:04:10.226940 13945 tablet_replica.cc:330] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0: stopping tablet replica
I20240503 14:04:10.229530 13945 raft_consensus.cc:2229] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.230139 13945 raft_consensus.cc:2258] T b2a8632a21724f14ab7ddd5dcee692df P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.232779 13945 tablet_replica.cc:330] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0: stopping tablet replica
I20240503 14:04:10.233505 13945 raft_consensus.cc:2229] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 14:04:10.228669 14649 consensus_peers.cc:485] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 -> Peer 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101): Couldn't send request to peer 186b18bf626043b39784c11cebd55db0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:10.234522 15015 tablet_copy_client.cc:308] T f023e4f1e70c47caad92f740e24b9acc P 1ef24c13b6b74446831df6680917bc9f: tablet copy: Beginning tablet copy session from remote peer at address 127.13.158.65:44101
I20240503 14:04:10.234488 15016 ts_tablet_manager.cc:902] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f: Initiating tablet copy from peer 36443f8bb0a844a7947756620254c2e0 (127.13.158.67:45667)
W20240503 14:04:10.237560 14649 consensus_peers.cc:485] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 -> Peer 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101): Couldn't send request to peer 186b18bf626043b39784c11cebd55db0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:10.234830 13945 raft_consensus.cc:2258] T 0dfef82a71c8452f99024d7e7edc988f P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.241482 13945 tablet_replica.cc:330] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0: stopping tablet replica
I20240503 14:04:10.242275 13945 raft_consensus.cc:2229] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 2 LEADER]: Raft consensus shutting down.
I20240503 14:04:10.243427 13945 raft_consensus.cc:2258] T 17937dd0f02348718b2704f4cddb0397 P 186b18bf626043b39784c11cebd55db0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.246119 13945 tablet_replica.cc:330] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0: stopping tablet replica
I20240503 14:04:10.247412 15012 tablet_copy_client.cc:308] T 99f1c0f5f2354dacbd45ac12c93e70bc P 1ef24c13b6b74446831df6680917bc9f: tablet copy: Beginning tablet copy session from remote peer at address 127.13.158.66:41235
I20240503 14:04:10.257290 13945 raft_consensus.cc:2229] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 2 LEADER]: Raft consensus shutting down.
I20240503 14:04:10.258798 13945 raft_consensus.cc:2258] T f023e4f1e70c47caad92f740e24b9acc P 186b18bf626043b39784c11cebd55db0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.263804 13945 tablet_replica.cc:330] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0: stopping tablet replica
I20240503 14:04:10.266885 13945 raft_consensus.cc:2229] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.267357 13945 raft_consensus.cc:2258] T 7759c34f658a42c5a42e9f8ca831fe07 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.269140 13945 tablet_replica.cc:330] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0: stopping tablet replica
I20240503 14:04:10.269631 13945 raft_consensus.cc:2229] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.270082 13945 raft_consensus.cc:2258] T d2302ac8c2e544c68cf11427f3bf9143 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.273180 13945 tablet_replica.cc:330] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0: stopping tablet replica
I20240503 14:04:10.273934 13945 raft_consensus.cc:2229] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.274652 13945 raft_consensus.cc:2258] T 1b3fa6230957409db8eff9dccbf62b32 P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.278041 13945 tablet_replica.cc:330] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0: stopping tablet replica
I20240503 14:04:10.278822 13945 raft_consensus.cc:2229] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.279810 13945 raft_consensus.cc:2258] T 99f1c0f5f2354dacbd45ac12c93e70bc P 186b18bf626043b39784c11cebd55db0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.281797 14696 tablet_copy_service.cc:140] P a7638e8e636a4601b99a740dd28fa895: Received BeginTabletCopySession request for tablet 99f1c0f5f2354dacbd45ac12c93e70bc from peer 1ef24c13b6b74446831df6680917bc9f ({username='slave'} at 127.0.0.1:52684)
I20240503 14:04:10.282513 14696 tablet_copy_service.cc:161] P a7638e8e636a4601b99a740dd28fa895: Beginning new tablet copy session on tablet 99f1c0f5f2354dacbd45ac12c93e70bc from peer 1ef24c13b6b74446831df6680917bc9f at {username='slave'} at 127.0.0.1:52684: session id = 1ef24c13b6b74446831df6680917bc9f-99f1c0f5f2354dacbd45ac12c93e70bc
W20240503 14:04:10.284078 14649 consensus_peers.cc:485] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 -> Peer 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101): Couldn't send request to peer 186b18bf626043b39784c11cebd55db0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:10.304311 14696 tablet_copy_source_session.cc:205] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:04:10.309722 15032 ts_tablet_manager.cc:902] T d2302ac8c2e544c68cf11427f3bf9143 P 1ef24c13b6b74446831df6680917bc9f: Initiating tablet copy from peer a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235)
I20240503 14:04:10.309959 15012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f1c0f5f2354dacbd45ac12c93e70bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:10.317492 15016 tablet_copy_client.cc:308] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f: tablet copy: Beginning tablet copy session from remote peer at address 127.13.158.67:45667
W20240503 14:04:10.324779 14868 connection.cc:535] client connection to 127.13.158.65:44101 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:04:10.325959 14870 connection.cc:441] server connection from 127.0.0.1:37796 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:37796 (request call id 1) could send its response
I20240503 14:04:10.339174 13945 tablet_server.cc:194] TabletServer@127.13.158.65:0 shutdown complete.
I20240503 14:04:10.350092 14770 tablet_copy_service.cc:140] P 36443f8bb0a844a7947756620254c2e0: Received BeginTabletCopySession request for tablet 1b3fa6230957409db8eff9dccbf62b32 from peer 1ef24c13b6b74446831df6680917bc9f ({username='slave'} at 127.0.0.1:44820)
I20240503 14:04:10.350778 14770 tablet_copy_service.cc:161] P 36443f8bb0a844a7947756620254c2e0: Beginning new tablet copy session on tablet 1b3fa6230957409db8eff9dccbf62b32 from peer 1ef24c13b6b74446831df6680917bc9f at {username='slave'} at 127.0.0.1:44820: session id = 1ef24c13b6b74446831df6680917bc9f-1b3fa6230957409db8eff9dccbf62b32
I20240503 14:04:10.359628 14770 tablet_copy_source_session.cc:205] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:04:10.363129 15016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3fa6230957409db8eff9dccbf62b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:10.364321 15012 tablet_copy_client.cc:783] T 99f1c0f5f2354dacbd45ac12c93e70bc P 1ef24c13b6b74446831df6680917bc9f: tablet copy: Starting download of 0 data blocks...
I20240503 14:04:10.365097 15012 tablet_copy_client.cc:647] T 99f1c0f5f2354dacbd45ac12c93e70bc P 1ef24c13b6b74446831df6680917bc9f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:04:10.368304 13945 tablet_server.cc:178] TabletServer@127.13.158.66:0 shutting down...
W20240503 14:04:10.373656 14557 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 14:04:10.374061 14557 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 14:04:10.380820 15016 tablet_copy_client.cc:783] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f: tablet copy: Starting download of 0 data blocks...
I20240503 14:04:10.381480 15016 tablet_copy_client.cc:647] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f: tablet copy: Starting download of 1 WAL segments...
W20240503 14:04:10.383155 14557 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 14:04:10.383543 14557 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 14:04:10.389241 15016 tablet_copy_client.cc:523] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:04:10.400187 15016 tablet_bootstrap.cc:492] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f: Bootstrap starting.
I20240503 14:04:10.402945 15032 tablet_copy_client.cc:308] T d2302ac8c2e544c68cf11427f3bf9143 P 1ef24c13b6b74446831df6680917bc9f: tablet copy: Beginning tablet copy session from remote peer at address 127.13.158.66:41235
I20240503 14:04:10.402958 13945 tablet_copy_service.cc:362] P a7638e8e636a4601b99a740dd28fa895: Destroying tablet copy session 1ef24c13b6b74446831df6680917bc9f-99f1c0f5f2354dacbd45ac12c93e70bc due to service shutdown
I20240503 14:04:10.403829 13945 tablet_copy_service.cc:433] P a7638e8e636a4601b99a740dd28fa895: ending tablet copy session 1ef24c13b6b74446831df6680917bc9f-99f1c0f5f2354dacbd45ac12c93e70bc on tablet 99f1c0f5f2354dacbd45ac12c93e70bc with peer 1ef24c13b6b74446831df6680917bc9f
W20240503 14:04:10.410915 14723 consensus_peers.cc:485] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 -> Peer 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101): Couldn't send request to peer 186b18bf626043b39784c11cebd55db0. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:44101: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:04:10.425608 14723 consensus_peers.cc:485] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 -> Peer 186b18bf626043b39784c11cebd55db0 (127.13.158.65:44101): Couldn't send request to peer 186b18bf626043b39784c11cebd55db0. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:44101: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:10.440583 15016 tablet_bootstrap.cc:492] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:04:10.441784 15016 tablet_bootstrap.cc:492] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f: Bootstrap complete.
I20240503 14:04:10.442152 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:10.442513 15016 ts_tablet_manager.cc:1367] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f: Time spent bootstrapping tablet: real 0.043s user 0.025s sys 0.004s
I20240503 14:04:10.444937 13945 tablet_replica.cc:330] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895: stopping tablet replica
I20240503 14:04:10.445814 13945 raft_consensus.cc:2229] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:10.445994 15016 raft_consensus.cc:348] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } }
I20240503 14:04:10.446909 13945 raft_consensus.cc:2258] T b2a8632a21724f14ab7ddd5dcee692df P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.446908 15016 raft_consensus.cc:729] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ef24c13b6b74446831df6680917bc9f, State: Initialized, Role: LEARNER
I20240503 14:04:10.447791 15016 consensus_queue.cc:260] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "186b18bf626043b39784c11cebd55db0" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 44101 } } peers { permanent_uuid: "a7638e8e636a4601b99a740dd28fa895" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "36443f8bb0a844a7947756620254c2e0" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 45667 } } peers { permanent_uuid: "1ef24c13b6b74446831df6680917bc9f" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 37341 } attrs { promote: true } }
I20240503 14:04:10.450021 13945 tablet_replica.cc:330] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895: stopping tablet replica
I20240503 14:04:10.450723 15016 ts_tablet_manager.cc:1398] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20240503 14:04:10.450888 13945 raft_consensus.cc:2229] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.451603 13945 raft_consensus.cc:2258] T 7759c34f658a42c5a42e9f8ca831fe07 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.454031 13945 tablet_replica.cc:330] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895: stopping tablet replica
I20240503 14:04:10.454842 13945 raft_consensus.cc:2229] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.455361 13945 raft_consensus.cc:2258] T 0dfef82a71c8452f99024d7e7edc988f P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.457443 13945 tablet_replica.cc:330] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895: stopping tablet replica
I20240503 14:04:10.457517 14770 tablet_copy_service.cc:342] P 36443f8bb0a844a7947756620254c2e0: Request end of tablet copy session 1ef24c13b6b74446831df6680917bc9f-1b3fa6230957409db8eff9dccbf62b32 received from {username='slave'} at 127.0.0.1:44820
I20240503 14:04:10.457994 14770 tablet_copy_service.cc:433] P 36443f8bb0a844a7947756620254c2e0: ending tablet copy session 1ef24c13b6b74446831df6680917bc9f-1b3fa6230957409db8eff9dccbf62b32 on tablet 1b3fa6230957409db8eff9dccbf62b32 with peer 1ef24c13b6b74446831df6680917bc9f
I20240503 14:04:10.458233 13945 raft_consensus.cc:2229] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.458828 13945 raft_consensus.cc:2258] T 1b3fa6230957409db8eff9dccbf62b32 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.461496 13945 tablet_replica.cc:330] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895: stopping tablet replica
I20240503 14:04:10.462292 13945 raft_consensus.cc:2229] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:10.463359 13945 raft_consensus.cc:2258] T d2302ac8c2e544c68cf11427f3bf9143 P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.465544 13945 tablet_replica.cc:330] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895: stopping tablet replica
I20240503 14:04:10.466050 13945 raft_consensus.cc:2229] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:10.467020 13945 raft_consensus.cc:2258] T 99f1c0f5f2354dacbd45ac12c93e70bc P a7638e8e636a4601b99a740dd28fa895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.469323 13945 tablet_replica.cc:330] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895: stopping tablet replica
I20240503 14:04:10.469796 13945 raft_consensus.cc:2229] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.470167 13945 raft_consensus.cc:2258] T 17937dd0f02348718b2704f4cddb0397 P a7638e8e636a4601b99a740dd28fa895 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.472219 13945 tablet_replica.cc:330] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895: stopping tablet replica
I20240503 14:04:10.472636 13945 raft_consensus.cc:2229] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.473078 13945 raft_consensus.cc:2258] T f023e4f1e70c47caad92f740e24b9acc P a7638e8e636a4601b99a740dd28fa895 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240503 14:04:10.494414 15012 ts_tablet_manager.cc:980] T 99f1c0f5f2354dacbd45ac12c93e70bc P 1ef24c13b6b74446831df6680917bc9f: Tablet Copy: Unable to fetch data from remote peer a7638e8e636a4601b99a740dd28fa895 (127.13.158.66:41235): Network error: Unable to download WAL segment with seq. number 1: unable to fetch data from remote: Client connection negotiation failed: client connection to 127.13.158.66:41235: connect: Connection refused (error 111)
I20240503 14:04:10.494880 15012 tablet_replica.cc:330] T 99f1c0f5f2354dacbd45ac12c93e70bc P 1ef24c13b6b74446831df6680917bc9f: stopping tablet replica
I20240503 14:04:10.495147 15012 raft_consensus.cc:2229] T 99f1c0f5f2354dacbd45ac12c93e70bc P 1ef24c13b6b74446831df6680917bc9f [term 1 LEARNER]: Raft consensus shutting down.
I20240503 14:04:10.495496 15012 raft_consensus.cc:2258] T 99f1c0f5f2354dacbd45ac12c93e70bc P 1ef24c13b6b74446831df6680917bc9f [term 1 LEARNER]: Raft consensus is shut down!
W20240503 14:04:10.497555 15012 tablet_copy_client.cc:1108] T 99f1c0f5f2354dacbd45ac12c93e70bc P 1ef24c13b6b74446831df6680917bc9f: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.13.158.66:41235: connect: Connection refused (error 111)
W20240503 14:04:10.498759 15012 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Already present: tried to load directory group for tablet 99f1c0f5f2354dacbd45ac12c93e70bc but one is already registered
I20240503 14:04:10.499213 15012 ts_tablet_manager.cc:1875] T 99f1c0f5f2354dacbd45ac12c93e70bc P 1ef24c13b6b74446831df6680917bc9f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:04:10.505336 14868 connection.cc:441] server connection from 127.0.0.1:37790 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:37790 (request call id 3) could send its response
I20240503 14:04:10.509541 13945 tablet_server.cc:194] TabletServer@127.13.158.66:0 shutdown complete.
I20240503 14:04:10.529979 13945 tablet_server.cc:178] TabletServer@127.13.158.67:0 shutting down...
W20240503 14:04:10.534780 14557 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 14:04:10.535033 14557 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 14:04:10.550973 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:10.551635 13945 tablet_replica.cc:330] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0: stopping tablet replica
I20240503 14:04:10.552232 13945 raft_consensus.cc:2229] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.552682 13945 raft_consensus.cc:2258] T 99f1c0f5f2354dacbd45ac12c93e70bc P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.555886 13945 tablet_replica.cc:330] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0: stopping tablet replica
I20240503 14:04:10.556442 13945 raft_consensus.cc:2229] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:10.557153 13945 raft_consensus.cc:2258] T 0dfef82a71c8452f99024d7e7edc988f P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.558815 13945 tablet_replica.cc:330] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0: stopping tablet replica
I20240503 14:04:10.559268 13945 raft_consensus.cc:2229] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.559641 13945 raft_consensus.cc:2258] T b2a8632a21724f14ab7ddd5dcee692df P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.561426 13945 tablet_replica.cc:330] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0: stopping tablet replica
I20240503 14:04:10.561915 13945 raft_consensus.cc:2229] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:10.562891 13945 raft_consensus.cc:2258] T 1b3fa6230957409db8eff9dccbf62b32 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.564971 13945 tablet_replica.cc:330] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0: stopping tablet replica
I20240503 14:04:10.565449 13945 raft_consensus.cc:2229] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.565860 13945 raft_consensus.cc:2258] T d2302ac8c2e544c68cf11427f3bf9143 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.567929 13945 tablet_replica.cc:330] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0: stopping tablet replica
I20240503 14:04:10.568437 13945 raft_consensus.cc:2229] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.568894 13945 raft_consensus.cc:2258] T f023e4f1e70c47caad92f740e24b9acc P 36443f8bb0a844a7947756620254c2e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.571106 13945 tablet_replica.cc:330] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0: stopping tablet replica
I20240503 14:04:10.571568 13945 raft_consensus.cc:2229] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:10.571967 13945 raft_consensus.cc:2258] T 17937dd0f02348718b2704f4cddb0397 P 36443f8bb0a844a7947756620254c2e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.573619 13945 tablet_replica.cc:330] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0: stopping tablet replica
I20240503 14:04:10.574054 13945 raft_consensus.cc:2229] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:10.574790 13945 raft_consensus.cc:2258] T 7759c34f658a42c5a42e9f8ca831fe07 P 36443f8bb0a844a7947756620254c2e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.601429 13945 tablet_server.cc:194] TabletServer@127.13.158.67:0 shutdown complete.
I20240503 14:04:10.619681 13945 tablet_server.cc:178] TabletServer@127.13.158.68:0 shutting down...
I20240503 14:04:10.642158 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:10.702996 15012 ts_tablet_manager.cc:1888] T 99f1c0f5f2354dacbd45ac12c93e70bc P 1ef24c13b6b74446831df6680917bc9f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.0
I20240503 14:04:10.703410 15012 log.cc:1196] T 99f1c0f5f2354dacbd45ac12c93e70bc P 1ef24c13b6b74446831df6680917bc9f: Deleting WAL directory at /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714745036514908-13945-0/minicluster-data/ts-3-root/wals/99f1c0f5f2354dacbd45ac12c93e70bc
I20240503 14:04:10.706080 13945 tablet_replica.cc:330] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f: stopping tablet replica
I20240503 14:04:10.706764 13945 raft_consensus.cc:2229] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f [term 1 LEARNER]: Raft consensus shutting down.
I20240503 14:04:10.707170 13945 raft_consensus.cc:2258] T 1b3fa6230957409db8eff9dccbf62b32 P 1ef24c13b6b74446831df6680917bc9f [term 1 LEARNER]: Raft consensus is shut down!
I20240503 14:04:10.727209 13945 tablet_server.cc:194] TabletServer@127.13.158.68:0 shutdown complete.
I20240503 14:04:10.735991 13945 master.cc:537] Master@127.13.158.126:39965 shutting down...
I20240503 14:04:10.752121 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:10.752635 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:10.752914 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 90b7767a39aa4fa09513935a8323e975: stopping tablet replica
I20240503 14:04:10.772212 13945 master.cc:552] Master@127.13.158.126:39965 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (9349 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240503 14:04:10.808992 13945 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.13.158.126:33787,127.13.158.125:45585,127.13.158.124:45841
I20240503 14:04:10.810303 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:10.816073 15045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:10.816859 15046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:10.817404 15048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:10.818024 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:10.818887 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:10.819068 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:10.819214 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745050819197 us; error 0 us; skew 500 ppm
I20240503 14:04:10.819640 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:10.822885 13945 webserver.cc:460] Webserver started at http://127.13.158.126:42443/ using document root <none> and password file <none>
I20240503 14:04:10.823367 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:10.823518 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:10.823751 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:10.824807 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-0-root/instance:
uuid: "1d951aa50a3140d09da925ec9541f796"
format_stamp: "Formatted at 2024-05-03 14:04:10 on dist-test-slave-jm3r"
I20240503 14:04:10.829371 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240503 14:04:10.832587 15053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:10.833318 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:04:10.833583 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-0-root
uuid: "1d951aa50a3140d09da925ec9541f796"
format_stamp: "Formatted at 2024-05-03 14:04:10 on dist-test-slave-jm3r"
I20240503 14:04:10.833838 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:10.849108 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:10.850306 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:10.887315 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.126:33787
I20240503 14:04:10.887415 15104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.126:33787 every 8 connection(s)
I20240503 14:04:10.891069 15105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:10.891186 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:10.897200 15107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:10.898003 15105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:04:10.898141 15108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:10.900265 15110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:10.903092 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:10.904039 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:10.904263 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:10.904436 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745050904422 us; error 0 us; skew 500 ppm
I20240503 14:04:10.904937 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:10.908771 13945 webserver.cc:460] Webserver started at http://127.13.158.125:41697/ using document root <none> and password file <none>
I20240503 14:04:10.909390 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:10.909583 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:10.909842 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:10.911233 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-1-root/instance:
uuid: "3bcbf4a560584845bad18bccf6f0539b"
format_stamp: "Formatted at 2024-05-03 14:04:10 on dist-test-slave-jm3r"
I20240503 14:04:10.912983 15105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:04:10.916636 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
W20240503 14:04:10.918355 15105 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.13.158.125:45585: Network error: Client connection negotiation failed: client connection to 127.13.158.125:45585: connect: Connection refused (error 111)
I20240503 14:04:10.921095 15118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:10.922235 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:04:10.922708 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-1-root
uuid: "3bcbf4a560584845bad18bccf6f0539b"
format_stamp: "Formatted at 2024-05-03 14:04:10 on dist-test-slave-jm3r"
I20240503 14:04:10.923035 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:10.935025 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:10.936206 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:10.974825 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.125:45585
I20240503 14:04:10.974961 15169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.125:45585 every 8 connection(s)
I20240503 14:04:10.974831 15105 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } attempt: 1
I20240503 14:04:10.979579 15171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:10.980142 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:10.986136 15173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:10.987537 15171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:04:10.988294 15174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:10.991186 15176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:10.992182 15105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:04:10.992933 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:10.993954 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:10.994205 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:10.994392 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745050994375 us; error 0 us; skew 500 ppm
I20240503 14:04:10.995101 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20240503 14:04:10.997121 15105 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.13.158.124:45841: Network error: Client connection negotiation failed: client connection to 127.13.158.124:45841: connect: Connection refused (error 111)
I20240503 14:04:10.998993 13945 webserver.cc:460] Webserver started at http://127.13.158.124:40753/ using document root <none> and password file <none>
I20240503 14:04:10.999531 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:10.999711 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:10.999979 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:11.000228 15171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:04:11.001060 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-2-root/instance:
uuid: "981df122ee70476f8c3b5f8a295a13f6"
format_stamp: "Formatted at 2024-05-03 14:04:10 on dist-test-slave-jm3r"
I20240503 14:04:11.006624 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:04:11.009991 15182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:11.010983 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:04:11.011451 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-2-root
uuid: "981df122ee70476f8c3b5f8a295a13f6"
format_stamp: "Formatted at 2024-05-03 14:04:10 on dist-test-slave-jm3r"
I20240503 14:04:11.011713 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:11.012727 15171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } has no permanent_uuid. Determining permanent_uuid...
W20240503 14:04:11.018661 15171 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.13.158.124:45841: Network error: Client connection negotiation failed: client connection to 127.13.158.124:45841: connect: Connection refused (error 111)
I20240503 14:04:11.022605 15105 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } attempt: 1
W20240503 14:04:11.026705 15105 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.13.158.124:45841: Network error: Client connection negotiation failed: client connection to 127.13.158.124:45841: connect: Connection refused (error 111)
I20240503 14:04:11.039141 15171 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } attempt: 1
W20240503 14:04:11.043244 15171 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.13.158.124:45841: Network error: Client connection negotiation failed: client connection to 127.13.158.124:45841: connect: Connection refused (error 111)
I20240503 14:04:11.043844 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:11.044896 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:11.061177 15105 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } attempt: 2
W20240503 14:04:11.065518 15105 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.13.158.124:45841: Network error: Client connection negotiation failed: client connection to 127.13.158.124:45841: connect: Connection refused (error 111)
I20240503 14:04:11.082988 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.124:45841
I20240503 14:04:11.083070 15234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.124:45841 every 8 connection(s)
I20240503 14:04:11.085856 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:04:11.086544 15235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:11.091941 15235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:04:11.101930 15235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:04:11.102627 15171 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } attempt: 2
I20240503 14:04:11.113307 15235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } has no permanent_uuid. Determining permanent_uuid...
I20240503 14:04:11.118577 15171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b: Bootstrap starting.
I20240503 14:04:11.123114 15171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:11.127172 15235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6: Bootstrap starting.
I20240503 14:04:11.127331 15171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b: No bootstrap required, opened a new log
I20240503 14:04:11.129312 15171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:11.129822 15171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:11.130053 15171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bcbf4a560584845bad18bccf6f0539b, State: Initialized, Role: FOLLOWER
I20240503 14:04:11.130640 15171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:11.131567 15235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:11.132473 15240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } } }
I20240503 14:04:11.133289 15240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:04:11.134050 15171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:04:11.135003 15105 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } attempt: 3
I20240503 14:04:11.136343 15235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6: No bootstrap required, opened a new log
I20240503 14:04:11.139072 15235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:11.139860 15235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:11.140183 15235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981df122ee70476f8c3b5f8a295a13f6, State: Initialized, Role: FOLLOWER
I20240503 14:04:11.141041 15235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:11.147902 15246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } } }
I20240503 14:04:11.148895 15246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:04:11.149442 15235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:04:11.153741 15105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796: Bootstrap starting.
W20240503 14:04:11.160162 15256 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240503 14:04:11.160411 15105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796: Neither blocks nor log segments found. Creating new log.
W20240503 14:04:11.160460 15256 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 14:04:11.166384 15105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796: No bootstrap required, opened a new log
I20240503 14:04:11.169220 15105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:11.169714 15105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:11.169972 15105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d951aa50a3140d09da925ec9541f796, State: Initialized, Role: FOLLOWER
I20240503 14:04:11.170576 15105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
W20240503 14:04:11.171802 15265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:04:11.172091 15265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 14:04:11.172349 15266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } } }
I20240503 14:04:11.173123 15266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:04:11.173794 15105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:04:11.183488 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240503 14:04:11.183720 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 14:04:11.185030 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:11.186272 15277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:04:11.186592 15277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 14:04:11.191197 15278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:11.192445 15279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:11.194061 15281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:11.194823 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:11.195511 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:11.195703 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:11.195852 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745051195837 us; error 0 us; skew 500 ppm
I20240503 14:04:11.196355 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:11.198704 13945 webserver.cc:460] Webserver started at http://127.13.158.65:41009/ using document root <none> and password file <none>
I20240503 14:04:11.199148 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:11.199314 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:11.199551 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:11.200659 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-0-root/instance:
uuid: "b279e7c1d74d4dffa12c660cee7ba74d"
format_stamp: "Formatted at 2024-05-03 14:04:11 on dist-test-slave-jm3r"
I20240503 14:04:11.204921 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:04:11.208326 15286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:11.209167 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:04:11.209441 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-0-root
uuid: "b279e7c1d74d4dffa12c660cee7ba74d"
format_stamp: "Formatted at 2024-05-03 14:04:11 on dist-test-slave-jm3r"
I20240503 14:04:11.209717 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:11.229821 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:11.231389 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:11.232782 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:11.235415 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:11.235641 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:11.235922 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:11.236089 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:11.266465 15240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:11.266919 15240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:11.268640 15240 leader_election.cc:288] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d951aa50a3140d09da925ec9541f796 (127.13.158.126:33787), 981df122ee70476f8c3b5f8a295a13f6 (127.13.158.124:45841)
I20240503 14:04:11.269500 15080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bcbf4a560584845bad18bccf6f0539b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d951aa50a3140d09da925ec9541f796" is_pre_election: true
I20240503 14:04:11.269851 15210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bcbf4a560584845bad18bccf6f0539b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981df122ee70476f8c3b5f8a295a13f6" is_pre_election: true
I20240503 14:04:11.270586 15210 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bcbf4a560584845bad18bccf6f0539b in term 0.
I20240503 14:04:11.270404 15080 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bcbf4a560584845bad18bccf6f0539b in term 0.
I20240503 14:04:11.272162 15122 leader_election.cc:302] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bcbf4a560584845bad18bccf6f0539b, 981df122ee70476f8c3b5f8a295a13f6; no voters:
I20240503 14:04:11.272926 15240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:11.273300 15240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:11.273641 15240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:11.278625 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.65:45435
I20240503 14:04:11.278788 15348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.65:45435 every 8 connection(s)
I20240503 14:04:11.285738 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:04:11.299883 15349 heartbeater.cc:347] Connected to a master server at 127.13.158.124:45841
I20240503 14:04:11.300307 15349 heartbeater.cc:464] Registering TS with master...
W20240503 14:04:11.300307 15356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:11.300765 15350 heartbeater.cc:347] Connected to a master server at 127.13.158.125:45585
I20240503 14:04:11.301174 15350 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:11.301529 15349 heartbeater.cc:511] Master 127.13.158.124:45841 requested a full tablet report, sending...
I20240503 14:04:11.301990 15350 heartbeater.cc:511] Master 127.13.158.125:45585 requested a full tablet report, sending...
I20240503 14:04:11.304430 15200 ts_manager.cc:194] Registered new tserver with Master: b279e7c1d74d4dffa12c660cee7ba74d (127.13.158.65:45435)
W20240503 14:04:11.304853 15357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:11.305637 15240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:11.307751 15240 leader_election.cc:288] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [CANDIDATE]: Term 1 election: Requested vote from peers 1d951aa50a3140d09da925ec9541f796 (127.13.158.126:33787), 981df122ee70476f8c3b5f8a295a13f6 (127.13.158.124:45841)
I20240503 14:04:11.309834 15351 heartbeater.cc:347] Connected to a master server at 127.13.158.126:33787
I20240503 14:04:11.310117 15135 ts_manager.cc:194] Registered new tserver with Master: b279e7c1d74d4dffa12c660cee7ba74d (127.13.158.65:45435)
I20240503 14:04:11.311692 15210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bcbf4a560584845bad18bccf6f0539b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981df122ee70476f8c3b5f8a295a13f6"
I20240503 14:04:11.311697 15080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bcbf4a560584845bad18bccf6f0539b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d951aa50a3140d09da925ec9541f796"
I20240503 14:04:11.312721 15080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:11.310266 15351 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:11.314381 15351 heartbeater.cc:511] Master 127.13.158.126:33787 requested a full tablet report, sending...
I20240503 14:04:11.312659 15210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:11.317348 13945 server_base.cc:1034] running on GCE node
W20240503 14:04:11.318696 15360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:11.319746 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:11.320024 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:11.320230 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745051320210 us; error 0 us; skew 500 ppm
I20240503 14:04:11.320729 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:11.323287 13945 webserver.cc:460] Webserver started at http://127.13.158.66:42581/ using document root <none> and password file <none>
I20240503 14:04:11.323740 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:11.323933 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:11.324378 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:11.325784 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-1-root/instance:
uuid: "1cc7f9e8e512410ab4010cd587cd75ea"
format_stamp: "Formatted at 2024-05-03 14:04:11 on dist-test-slave-jm3r"
I20240503 14:04:11.330327 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:04:11.333730 15364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:11.334647 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:04:11.334928 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-1-root
uuid: "1cc7f9e8e512410ab4010cd587cd75ea"
format_stamp: "Formatted at 2024-05-03 14:04:11 on dist-test-slave-jm3r"
I20240503 14:04:11.335199 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:11.344121 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:11.345297 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:11.346724 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:11.349058 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:11.349272 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:11.349506 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:11.349656 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:11.388260 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.66:44869
I20240503 14:04:11.388353 15426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.66:44869 every 8 connection(s)
I20240503 14:04:11.402980 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:04:11.406590 15427 heartbeater.cc:347] Connected to a master server at 127.13.158.124:45841
I20240503 14:04:11.407039 15427 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:11.407966 15427 heartbeater.cc:511] Master 127.13.158.124:45841 requested a full tablet report, sending...
I20240503 14:04:11.408174 15428 heartbeater.cc:347] Connected to a master server at 127.13.158.125:45585
I20240503 14:04:11.408635 15428 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:11.409592 15428 heartbeater.cc:511] Master 127.13.158.125:45585 requested a full tablet report, sending...
I20240503 14:04:11.410778 15080 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bcbf4a560584845bad18bccf6f0539b in term 1.
I20240503 14:04:11.411883 15210 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bcbf4a560584845bad18bccf6f0539b in term 1.
I20240503 14:04:11.414404 15070 ts_manager.cc:194] Registered new tserver with Master: b279e7c1d74d4dffa12c660cee7ba74d (127.13.158.65:45435)
I20240503 14:04:11.414579 15122 leader_election.cc:302] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bcbf4a560584845bad18bccf6f0539b, 981df122ee70476f8c3b5f8a295a13f6; no voters:
I20240503 14:04:11.415468 15240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:11.417529 15200 ts_manager.cc:194] Registered new tserver with Master: 1cc7f9e8e512410ab4010cd587cd75ea (127.13.158.66:44869)
I20240503 14:04:11.420892 15429 heartbeater.cc:347] Connected to a master server at 127.13.158.126:33787
I20240503 14:04:11.421348 15429 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:11.421486 15240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 1 LEADER]: Becoming Leader. State: Replica: 3bcbf4a560584845bad18bccf6f0539b, State: Running, Role: LEADER
I20240503 14:04:11.422410 15429 heartbeater.cc:511] Master 127.13.158.126:33787 requested a full tablet report, sending...
I20240503 14:04:11.422515 15240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
W20240503 14:04:11.424074 15434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:11.425827 15070 ts_manager.cc:194] Registered new tserver with Master: 1cc7f9e8e512410ab4010cd587cd75ea (127.13.158.66:44869)
W20240503 14:04:11.427106 15435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:11.430601 15438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:11.432158 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:11.433034 15135 ts_manager.cc:194] Registered new tserver with Master: 1cc7f9e8e512410ab4010cd587cd75ea (127.13.158.66:44869)
I20240503 14:04:11.432158 15436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bcbf4a560584845bad18bccf6f0539b. Latest consensus state: current_term: 1 leader_uuid: "3bcbf4a560584845bad18bccf6f0539b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } } }
I20240503 14:04:11.433369 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 14:04:11.433534 15436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [sys.catalog]: This master's current role is: LEADER
W20240503 14:04:11.433709 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:11.433961 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745051433940 us; error 0 us; skew 500 ppm
I20240503 14:04:11.434791 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:11.435351 15441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:04:11.437844 13945 webserver.cc:460] Webserver started at http://127.13.158.67:42471/ using document root <none> and password file <none>
I20240503 14:04:11.438326 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:11.438540 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:11.438793 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:11.439823 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-2-root/instance:
uuid: "7bf5bbb9a5764ce6aa4efbfb12fe448f"
format_stamp: "Formatted at 2024-05-03 14:04:11 on dist-test-slave-jm3r"
I20240503 14:04:11.440356 15441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:04:11.444795 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20240503 14:04:11.448709 15210 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 1 FOLLOWER]: Refusing update from remote peer 3bcbf4a560584845bad18bccf6f0539b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:04:11.448709 15080 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 1 FOLLOWER]: Refusing update from remote peer 3bcbf4a560584845bad18bccf6f0539b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:04:11.450389 15446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:11.450630 15240 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [LEADER]: Connected to new peer: Peer: permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:11.451665 13945 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20240503 14:04:11.451390 15436 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:11.452558 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-2-root
uuid: "7bf5bbb9a5764ce6aa4efbfb12fe448f"
format_stamp: "Formatted at 2024-05-03 14:04:11 on dist-test-slave-jm3r"
I20240503 14:04:11.452881 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714745036514908-13945-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:11.464534 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:11.465673 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:11.468178 15246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bcbf4a560584845bad18bccf6f0539b. Latest consensus state: current_term: 1 leader_uuid: "3bcbf4a560584845bad18bccf6f0539b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } } }
I20240503 14:04:11.469009 15246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:04:11.469362 15266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bcbf4a560584845bad18bccf6f0539b. Latest consensus state: current_term: 1 leader_uuid: "3bcbf4a560584845bad18bccf6f0539b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } } }
I20240503 14:04:11.469905 15266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:04:11.470374 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:11.480971 15266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bcbf4a560584845bad18bccf6f0539b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } } }
I20240503 14:04:11.481843 15266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:04:11.484851 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:11.485220 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:11.486398 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:11.486660 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.002s
I20240503 14:04:11.489601 15441 catalog_manager.cc:1347] Generated new cluster ID: 8b40b1ab28cc444e91019db01cd40919
I20240503 14:04:11.489943 15441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:04:11.490545 15246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bcbf4a560584845bad18bccf6f0539b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } } }
I20240503 14:04:11.491354 15246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 14:04:11.490572 15240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bcbf4a560584845bad18bccf6f0539b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } } }
I20240503 14:04:11.493095 15240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:11.493633 15240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bcbf4a560584845bad18bccf6f0539b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } } }
I20240503 14:04:11.494313 15240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:11.528381 15441 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:04:11.530056 15441 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:04:11.554466 15441 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b: Generated new TSK 0
I20240503 14:04:11.555233 15441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:04:11.561313 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.67:35741
I20240503 14:04:11.561396 15517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.67:35741 every 8 connection(s)
I20240503 14:04:11.581517 15518 heartbeater.cc:347] Connected to a master server at 127.13.158.124:45841
I20240503 14:04:11.581982 15518 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:11.582811 15518 heartbeater.cc:511] Master 127.13.158.124:45841 requested a full tablet report, sending...
I20240503 14:04:11.584244 15520 heartbeater.cc:347] Connected to a master server at 127.13.158.126:33787
I20240503 14:04:11.584661 15520 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:11.585508 15200 ts_manager.cc:194] Registered new tserver with Master: 7bf5bbb9a5764ce6aa4efbfb12fe448f (127.13.158.67:35741)
I20240503 14:04:11.586099 15520 heartbeater.cc:511] Master 127.13.158.126:33787 requested a full tablet report, sending...
I20240503 14:04:11.587231 15519 heartbeater.cc:347] Connected to a master server at 127.13.158.125:45585
I20240503 14:04:11.587535 15519 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:11.588284 15519 heartbeater.cc:511] Master 127.13.158.125:45585 requested a full tablet report, sending...
I20240503 14:04:11.588549 15070 ts_manager.cc:194] Registered new tserver with Master: 7bf5bbb9a5764ce6aa4efbfb12fe448f (127.13.158.67:35741)
I20240503 14:04:11.589990 15135 ts_manager.cc:194] Registered new tserver with Master: 7bf5bbb9a5764ce6aa4efbfb12fe448f (127.13.158.67:35741)
I20240503 14:04:11.590629 13945 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.010360369s
I20240503 14:04:11.591699 15135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43518
I20240503 14:04:12.175765 15265 catalog_manager.cc:1259] Loaded cluster ID: 8b40b1ab28cc444e91019db01cd40919
I20240503 14:04:12.176046 15265 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6: loading cluster ID for follower catalog manager: success
I20240503 14:04:12.180358 15265 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6: acquiring CA information for follower catalog manager: success
I20240503 14:04:12.183571 15265 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 14:04:12.190171 15277 catalog_manager.cc:1259] Loaded cluster ID: 8b40b1ab28cc444e91019db01cd40919
I20240503 14:04:12.190498 15277 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796: loading cluster ID for follower catalog manager: success
I20240503 14:04:12.194718 15277 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796: acquiring CA information for follower catalog manager: success
I20240503 14:04:12.197679 15277 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 14:04:12.315025 15135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43494
I20240503 14:04:12.437382 15135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43504
I20240503 14:04:12.593966 15519 heartbeater.cc:503] Master 127.13.158.125:45585 was elected leader, sending a full tablet report...
I20240503 14:04:13.156824 15254 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:13.317534 15350 heartbeater.cc:503] Master 127.13.158.125:45585 was elected leader, sending a full tablet report...
I20240503 14:04:13.439666 15428 heartbeater.cc:503] Master 127.13.158.125:45585 was elected leader, sending a full tablet report...
I20240503 14:04:13.665098 13945 test_util.cc:268] Using random seed: 835573543
I20240503 14:04:13.703455 15135 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43526:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:04:13.770545 15314 tablet_service.cc:1467] Processing CreateTablet for tablet 948a16c60541446e9371b05b7c2e8b2b (DEFAULT_TABLE table=test-workload [id=bf62f5e964ad4b828a3508b6fbe21a9b]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 14:04:13.770990 15483 tablet_service.cc:1467] Processing CreateTablet for tablet de853b8940c746bd85f8767668a161f7 (DEFAULT_TABLE table=test-workload [id=bf62f5e964ad4b828a3508b6fbe21a9b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 14:04:13.771821 15314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948a16c60541446e9371b05b7c2e8b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:13.772337 15483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de853b8940c746bd85f8767668a161f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:13.774468 15392 tablet_service.cc:1467] Processing CreateTablet for tablet 219425137e4f43f286192401cd7a7a49 (DEFAULT_TABLE table=test-workload [id=bf62f5e964ad4b828a3508b6fbe21a9b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 14:04:13.775905 15392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219425137e4f43f286192401cd7a7a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:13.788806 15554 tablet_bootstrap.cc:492] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d: Bootstrap starting.
I20240503 14:04:13.791875 15555 tablet_bootstrap.cc:492] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea: Bootstrap starting.
I20240503 14:04:13.792822 15553 tablet_bootstrap.cc:492] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f: Bootstrap starting.
I20240503 14:04:13.794795 15554 tablet_bootstrap.cc:654] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:13.798671 15553 tablet_bootstrap.cc:654] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:13.798671 15555 tablet_bootstrap.cc:654] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:13.799896 15554 tablet_bootstrap.cc:492] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d: No bootstrap required, opened a new log
I20240503 14:04:13.800292 15554 ts_tablet_manager.cc:1367] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240503 14:04:13.802429 15554 raft_consensus.cc:348] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b279e7c1d74d4dffa12c660cee7ba74d" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 45435 } }
I20240503 14:04:13.803081 15554 raft_consensus.cc:374] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:13.803382 15554 raft_consensus.cc:729] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b279e7c1d74d4dffa12c660cee7ba74d, State: Initialized, Role: FOLLOWER
I20240503 14:04:13.804139 15554 consensus_queue.cc:260] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b279e7c1d74d4dffa12c660cee7ba74d" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 45435 } }
I20240503 14:04:13.804687 15554 raft_consensus.cc:388] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:13.804791 15555 tablet_bootstrap.cc:492] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea: No bootstrap required, opened a new log
I20240503 14:04:13.804980 15554 raft_consensus.cc:482] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:13.805347 15554 raft_consensus.cc:3037] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:13.805629 15553 tablet_bootstrap.cc:492] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f: No bootstrap required, opened a new log
I20240503 14:04:13.806068 15553 ts_tablet_manager.cc:1367] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20240503 14:04:13.805318 15555 ts_tablet_manager.cc:1367] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20240503 14:04:13.808773 15553 raft_consensus.cc:348] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf5bbb9a5764ce6aa4efbfb12fe448f" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35741 } }
I20240503 14:04:13.809365 15553 raft_consensus.cc:374] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:13.809492 15555 raft_consensus.cc:348] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc7f9e8e512410ab4010cd587cd75ea" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44869 } }
I20240503 14:04:13.809698 15553 raft_consensus.cc:729] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf5bbb9a5764ce6aa4efbfb12fe448f, State: Initialized, Role: FOLLOWER
I20240503 14:04:13.810101 15555 raft_consensus.cc:374] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:13.810483 15555 raft_consensus.cc:729] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc7f9e8e512410ab4010cd587cd75ea, State: Initialized, Role: FOLLOWER
I20240503 14:04:13.810523 15553 consensus_queue.cc:260] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf5bbb9a5764ce6aa4efbfb12fe448f" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35741 } }
I20240503 14:04:13.811113 15554 raft_consensus.cc:504] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b279e7c1d74d4dffa12c660cee7ba74d" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 45435 } }
I20240503 14:04:13.811409 15553 raft_consensus.cc:388] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:13.811492 15555 consensus_queue.cc:260] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc7f9e8e512410ab4010cd587cd75ea" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44869 } }
I20240503 14:04:13.811825 15554 leader_election.cc:302] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b279e7c1d74d4dffa12c660cee7ba74d; no voters:
I20240503 14:04:13.812336 15555 raft_consensus.cc:388] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:13.812702 15555 raft_consensus.cc:482] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:13.811879 15553 raft_consensus.cc:482] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:13.813047 15555 raft_consensus.cc:3037] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:13.813181 15553 raft_consensus.cc:3037] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:13.817667 15553 raft_consensus.cc:504] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf5bbb9a5764ce6aa4efbfb12fe448f" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35741 } }
I20240503 14:04:13.818305 15553 leader_election.cc:302] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bf5bbb9a5764ce6aa4efbfb12fe448f; no voters:
I20240503 14:04:13.818589 15554 leader_election.cc:288] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:13.819058 15559 raft_consensus.cc:2781] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:13.820092 15555 raft_consensus.cc:504] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc7f9e8e512410ab4010cd587cd75ea" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44869 } }
I20240503 14:04:13.820953 15555 leader_election.cc:302] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cc7f9e8e512410ab4010cd587cd75ea; no voters:
I20240503 14:04:13.826489 15560 raft_consensus.cc:2781] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:13.827811 15553 leader_election.cc:288] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:13.830581 15554 ts_tablet_manager.cc:1398] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d: Time spent starting tablet: real 0.030s user 0.023s sys 0.009s
I20240503 14:04:13.832994 15559 raft_consensus.cc:686] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [term 1 LEADER]: Becoming Leader. State: Replica: b279e7c1d74d4dffa12c660cee7ba74d, State: Running, Role: LEADER
I20240503 14:04:13.833873 15559 consensus_queue.cc:237] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b279e7c1d74d4dffa12c660cee7ba74d" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 45435 } }
I20240503 14:04:13.834635 15553 ts_tablet_manager.cc:1398] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f: Time spent starting tablet: real 0.028s user 0.020s sys 0.000s
I20240503 14:04:13.837211 15560 raft_consensus.cc:686] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [term 1 LEADER]: Becoming Leader. State: Replica: 7bf5bbb9a5764ce6aa4efbfb12fe448f, State: Running, Role: LEADER
I20240503 14:04:13.838157 15560 consensus_queue.cc:237] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf5bbb9a5764ce6aa4efbfb12fe448f" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35741 } }
I20240503 14:04:13.840696 15555 leader_election.cc:288] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:13.841104 15563 raft_consensus.cc:2781] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:13.849756 15563 raft_consensus.cc:686] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [term 1 LEADER]: Becoming Leader. State: Replica: 1cc7f9e8e512410ab4010cd587cd75ea, State: Running, Role: LEADER
I20240503 14:04:13.850791 15563 consensus_queue.cc:237] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc7f9e8e512410ab4010cd587cd75ea" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44869 } }
I20240503 14:04:13.862910 15555 ts_tablet_manager.cc:1398] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea: Time spent starting tablet: real 0.056s user 0.028s sys 0.023s
I20240503 14:04:13.870693 15135 catalog_manager.cc:5484] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bf5bbb9a5764ce6aa4efbfb12fe448f (127.13.158.67). New cstate: current_term: 1 leader_uuid: "7bf5bbb9a5764ce6aa4efbfb12fe448f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf5bbb9a5764ce6aa4efbfb12fe448f" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35741 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:13.871944 15134 catalog_manager.cc:5484] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d reported cstate change: term changed from 0 to 1, leader changed from <none> to b279e7c1d74d4dffa12c660cee7ba74d (127.13.158.65). New cstate: current_term: 1 leader_uuid: "b279e7c1d74d4dffa12c660cee7ba74d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b279e7c1d74d4dffa12c660cee7ba74d" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 45435 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:13.901324 15133 catalog_manager.cc:5484] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc7f9e8e512410ab4010cd587cd75ea (127.13.158.66). New cstate: current_term: 1 leader_uuid: "1cc7f9e8e512410ab4010cd587cd75ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc7f9e8e512410ab4010cd587cd75ea" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44869 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:13.945977 13945 master.cc:537] Master@127.13.158.125:45585 shutting down...
I20240503 14:04:13.962996 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:13.963848 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:13.964170 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3bcbf4a560584845bad18bccf6f0539b: stopping tablet replica
I20240503 14:04:13.984938 13945 master.cc:552] Master@127.13.158.125:45585 shutdown complete.
W20240503 14:04:14.925525 15350 heartbeater.cc:644] Failed to heartbeat to 127.13.158.125:45585 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.158.125:45585: connect: Connection refused (error 111)
I20240503 14:04:15.442849 15578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3bcbf4a560584845bad18bccf6f0539b)
I20240503 14:04:15.443343 15578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:15.444921 15579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3bcbf4a560584845bad18bccf6f0539b)
I20240503 14:04:15.445116 15578 leader_election.cc:288] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bcbf4a560584845bad18bccf6f0539b (127.13.158.125:45585), 981df122ee70476f8c3b5f8a295a13f6 (127.13.158.124:45841)
I20240503 14:04:15.445516 15579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:15.446348 15210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d951aa50a3140d09da925ec9541f796" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "981df122ee70476f8c3b5f8a295a13f6" is_pre_election: true
I20240503 14:04:15.447052 15210 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d951aa50a3140d09da925ec9541f796 in term 1.
I20240503 14:04:15.448223 15579 leader_election.cc:288] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d951aa50a3140d09da925ec9541f796 (127.13.158.126:33787), 3bcbf4a560584845bad18bccf6f0539b (127.13.158.125:45585)
W20240503 14:04:15.448500 15056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.158.125:45585: connect: Connection refused (error 111) [suppressed 17 similar messages]
I20240503 14:04:15.448505 15079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "981df122ee70476f8c3b5f8a295a13f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "1d951aa50a3140d09da925ec9541f796" is_pre_election: true
I20240503 14:04:15.449424 15057 leader_election.cc:302] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d951aa50a3140d09da925ec9541f796, 981df122ee70476f8c3b5f8a295a13f6; no voters:
I20240503 14:04:15.449935 15079 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 981df122ee70476f8c3b5f8a295a13f6 in term 1.
I20240503 14:04:15.451414 15578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:04:15.452937 15183 leader_election.cc:302] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d951aa50a3140d09da925ec9541f796, 981df122ee70476f8c3b5f8a295a13f6; no voters:
I20240503 14:04:15.453292 15578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3bcbf4a560584845bad18bccf6f0539b)
W20240503 14:04:15.453437 15056 leader_election.cc:334] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3bcbf4a560584845bad18bccf6f0539b (127.13.158.125:45585): Network error: Client connection negotiation failed: client connection to 127.13.158.125:45585: connect: Connection refused (error 111)
I20240503 14:04:15.453939 15579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:04:15.453886 15578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:04:15.453869 15185 leader_election.cc:334] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3bcbf4a560584845bad18bccf6f0539b (127.13.158.125:45585): Network error: Client connection negotiation failed: client connection to 127.13.158.125:45585: connect: Connection refused (error 111)
I20240503 14:04:15.454569 15579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3bcbf4a560584845bad18bccf6f0539b)
I20240503 14:04:15.454989 15579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:15.458946 15578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:15.459295 15579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:15.460462 15578 leader_election.cc:288] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [CANDIDATE]: Term 2 election: Requested vote from peers 3bcbf4a560584845bad18bccf6f0539b (127.13.158.125:45585), 981df122ee70476f8c3b5f8a295a13f6 (127.13.158.124:45841)
I20240503 14:04:15.461481 15210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d951aa50a3140d09da925ec9541f796" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "981df122ee70476f8c3b5f8a295a13f6"
I20240503 14:04:15.461582 15079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "981df122ee70476f8c3b5f8a295a13f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "1d951aa50a3140d09da925ec9541f796"
I20240503 14:04:15.462352 15210 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1d951aa50a3140d09da925ec9541f796 in current term 2: Already voted for candidate 981df122ee70476f8c3b5f8a295a13f6 in this term.
I20240503 14:04:15.462476 15079 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 981df122ee70476f8c3b5f8a295a13f6 in current term 2: Already voted for candidate 1d951aa50a3140d09da925ec9541f796 in this term.
I20240503 14:04:15.461624 15579 leader_election.cc:288] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [CANDIDATE]: Term 2 election: Requested vote from peers 1d951aa50a3140d09da925ec9541f796 (127.13.158.126:33787), 3bcbf4a560584845bad18bccf6f0539b (127.13.158.125:45585)
W20240503 14:04:15.464666 15056 leader_election.cc:334] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3bcbf4a560584845bad18bccf6f0539b (127.13.158.125:45585): Network error: Client connection negotiation failed: client connection to 127.13.158.125:45585: connect: Connection refused (error 111)
I20240503 14:04:15.465122 15056 leader_election.cc:302] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d951aa50a3140d09da925ec9541f796; no voters: 3bcbf4a560584845bad18bccf6f0539b, 981df122ee70476f8c3b5f8a295a13f6
I20240503 14:04:15.465880 15578 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240503 14:04:15.466135 15185 leader_election.cc:334] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3bcbf4a560584845bad18bccf6f0539b (127.13.158.125:45585): Network error: Client connection negotiation failed: client connection to 127.13.158.125:45585: connect: Connection refused (error 111)
I20240503 14:04:15.466622 15185 leader_election.cc:302] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 981df122ee70476f8c3b5f8a295a13f6; no voters: 1d951aa50a3140d09da925ec9541f796, 3bcbf4a560584845bad18bccf6f0539b
I20240503 14:04:15.467278 15579 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 14:04:16.986510 15591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:16.987166 15591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:16.988916 15591 leader_election.cc:288] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1d951aa50a3140d09da925ec9541f796 (127.13.158.126:33787), 3bcbf4a560584845bad18bccf6f0539b (127.13.158.125:45585)
I20240503 14:04:16.990312 15079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "981df122ee70476f8c3b5f8a295a13f6" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "1d951aa50a3140d09da925ec9541f796" is_pre_election: true
I20240503 14:04:16.991330 15079 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 981df122ee70476f8c3b5f8a295a13f6 in term 2.
I20240503 14:04:16.993019 15183 leader_election.cc:302] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d951aa50a3140d09da925ec9541f796, 981df122ee70476f8c3b5f8a295a13f6; no voters:
I20240503 14:04:16.993803 15591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:04:16.994144 15591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:16.994407 15591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 2 FOLLOWER]: Advancing to term 3
W20240503 14:04:16.995177 15185 leader_election.cc:334] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3bcbf4a560584845bad18bccf6f0539b (127.13.158.125:45585): Network error: Client connection negotiation failed: client connection to 127.13.158.125:45585: connect: Connection refused (error 111)
I20240503 14:04:17.000531 15591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:17.002373 15591 leader_election.cc:288] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [CANDIDATE]: Term 3 election: Requested vote from peers 1d951aa50a3140d09da925ec9541f796 (127.13.158.126:33787), 3bcbf4a560584845bad18bccf6f0539b (127.13.158.125:45585)
I20240503 14:04:17.003228 15079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "981df122ee70476f8c3b5f8a295a13f6" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "1d951aa50a3140d09da925ec9541f796"
I20240503 14:04:17.003701 15079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 2 FOLLOWER]: Advancing to term 3
W20240503 14:04:17.005793 15185 leader_election.cc:334] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3bcbf4a560584845bad18bccf6f0539b (127.13.158.125:45585): Network error: Client connection negotiation failed: client connection to 127.13.158.125:45585: connect: Connection refused (error 111)
I20240503 14:04:17.008924 15079 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 981df122ee70476f8c3b5f8a295a13f6 in term 3.
I20240503 14:04:17.009879 15183 leader_election.cc:302] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d951aa50a3140d09da925ec9541f796, 981df122ee70476f8c3b5f8a295a13f6; no voters: 3bcbf4a560584845bad18bccf6f0539b
I20240503 14:04:17.010577 15591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:17.011689 15591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 3 LEADER]: Becoming Leader. State: Replica: 981df122ee70476f8c3b5f8a295a13f6, State: Running, Role: LEADER
I20240503 14:04:17.012481 15591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } }
I20240503 14:04:17.016342 15594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 981df122ee70476f8c3b5f8a295a13f6. Latest consensus state: current_term: 3 leader_uuid: "981df122ee70476f8c3b5f8a295a13f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d951aa50a3140d09da925ec9541f796" member_type: VOTER last_known_addr { host: "127.13.158.126" port: 33787 } } peers { permanent_uuid: "3bcbf4a560584845bad18bccf6f0539b" member_type: VOTER last_known_addr { host: "127.13.158.125" port: 45585 } } peers { permanent_uuid: "981df122ee70476f8c3b5f8a295a13f6" member_type: VOTER last_known_addr { host: "127.13.158.124" port: 45841 } } }
I20240503 14:04:17.017306 15594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:17.018853 15596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:04:17.024471 15596 catalog_manager.cc:669] Loaded metadata for table test-workload [id=bf62f5e964ad4b828a3508b6fbe21a9b]
I20240503 14:04:17.030187 15596 tablet_loader.cc:96] loaded metadata for tablet 219425137e4f43f286192401cd7a7a49 (table test-workload [id=bf62f5e964ad4b828a3508b6fbe21a9b])
I20240503 14:04:17.031291 15596 tablet_loader.cc:96] loaded metadata for tablet 948a16c60541446e9371b05b7c2e8b2b (table test-workload [id=bf62f5e964ad4b828a3508b6fbe21a9b])
I20240503 14:04:17.032689 15596 tablet_loader.cc:96] loaded metadata for tablet de853b8940c746bd85f8767668a161f7 (table test-workload [id=bf62f5e964ad4b828a3508b6fbe21a9b])
I20240503 14:04:17.033946 15596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:04:17.037263 15596 catalog_manager.cc:1259] Loaded cluster ID: 8b40b1ab28cc444e91019db01cd40919
I20240503 14:04:17.037560 15596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:04:17.041092 15596 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:04:17.044171 15596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6: Loaded TSK: 0
I20240503 14:04:17.045158 15596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:04:17.170320 15264 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:17.170765 15264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:17.201196 13945 tablet_server.cc:178] TabletServer@127.13.158.65:0 shutting down...
I20240503 14:04:17.234534 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:17.235507 13945 tablet_replica.cc:330] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d: stopping tablet replica
I20240503 14:04:17.236202 13945 raft_consensus.cc:2229] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:17.236750 13945 raft_consensus.cc:2258] T 948a16c60541446e9371b05b7c2e8b2b P b279e7c1d74d4dffa12c660cee7ba74d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:17.256991 13945 tablet_server.cc:194] TabletServer@127.13.158.65:0 shutdown complete.
I20240503 14:04:17.268447 13945 tablet_server.cc:178] TabletServer@127.13.158.66:0 shutting down...
I20240503 14:04:17.291417 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:17.292106 13945 tablet_replica.cc:330] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea: stopping tablet replica
I20240503 14:04:17.292606 13945 raft_consensus.cc:2229] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:17.293067 13945 raft_consensus.cc:2258] T 219425137e4f43f286192401cd7a7a49 P 1cc7f9e8e512410ab4010cd587cd75ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:17.311786 13945 tablet_server.cc:194] TabletServer@127.13.158.66:0 shutdown complete.
I20240503 14:04:17.322618 13945 tablet_server.cc:178] TabletServer@127.13.158.67:0 shutting down...
I20240503 14:04:17.345276 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:17.345947 13945 tablet_replica.cc:330] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f: stopping tablet replica
I20240503 14:04:17.346501 13945 raft_consensus.cc:2229] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:17.346947 13945 raft_consensus.cc:2258] T de853b8940c746bd85f8767668a161f7 P 7bf5bbb9a5764ce6aa4efbfb12fe448f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:17.365023 13945 tablet_server.cc:194] TabletServer@127.13.158.67:0 shutdown complete.
I20240503 14:04:17.374712 13945 master.cc:537] Master@127.13.158.126:33787 shutting down...
I20240503 14:04:17.395499 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:17.396020 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:17.396319 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1d951aa50a3140d09da925ec9541f796: stopping tablet replica
I20240503 14:04:17.414983 13945 master.cc:552] Master@127.13.158.126:33787 shutdown complete.
I20240503 14:04:17.429530 13945 master.cc:537] Master@127.13.158.124:45841 shutting down...
I20240503 14:04:17.444336 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 3 LEADER]: Raft consensus shutting down.
I20240503 14:04:17.444993 13945 pending_rounds.cc:62] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6: Trying to abort 1 pending ops.
I20240503 14:04:17.445191 13945 pending_rounds.cc:68] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6: Aborting op as it isn't in flight: id { term: 3 index: 11 } timestamp: 7023595753529712640 op_type: NO_OP noop_request { }
I20240503 14:04:17.445560 13945 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:04:17.445807 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:17.446193 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 981df122ee70476f8c3b5f8a295a13f6: stopping tablet replica
I20240503 14:04:17.465319 13945 master.cc:552] Master@127.13.158.124:45841 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6688 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240503 14:04:17.498042 13945 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.13.158.126:46369
I20240503 14:04:17.499132 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:17.505772 15597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:17.506585 15598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:17.507086 15600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:17.508217 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:17.509261 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:17.509498 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:17.509649 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745057509633 us; error 0 us; skew 500 ppm
I20240503 14:04:17.510178 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:17.517398 13945 webserver.cc:460] Webserver started at http://127.13.158.126:37201/ using document root <none> and password file <none>
I20240503 14:04:17.517910 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:17.518079 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:17.518366 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:17.519655 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/master-0-root/instance:
uuid: "47a15a0b83274f2e96245743094d3bd0"
format_stamp: "Formatted at 2024-05-03 14:04:17 on dist-test-slave-jm3r"
I20240503 14:04:17.524113 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:04:17.527585 15605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:17.528406 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:04:17.528689 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/master-0-root
uuid: "47a15a0b83274f2e96245743094d3bd0"
format_stamp: "Formatted at 2024-05-03 14:04:17 on dist-test-slave-jm3r"
I20240503 14:04:17.528946 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:17.557390 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:17.558701 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:17.595072 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.126:46369
I20240503 14:04:17.595175 15656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.126:46369 every 8 connection(s)
I20240503 14:04:17.598747 15657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:17.609052 15657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0: Bootstrap starting.
I20240503 14:04:17.613154 15657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:17.617151 15657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0: No bootstrap required, opened a new log
I20240503 14:04:17.619078 15657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a15a0b83274f2e96245743094d3bd0" member_type: VOTER }
I20240503 14:04:17.619455 15657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:17.619642 15657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a15a0b83274f2e96245743094d3bd0, State: Initialized, Role: FOLLOWER
I20240503 14:04:17.620133 15657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a15a0b83274f2e96245743094d3bd0" member_type: VOTER }
I20240503 14:04:17.620596 15657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:17.620783 15657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:17.620993 15657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:17.625280 15657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a15a0b83274f2e96245743094d3bd0" member_type: VOTER }
I20240503 14:04:17.625783 15657 leader_election.cc:302] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47a15a0b83274f2e96245743094d3bd0; no voters:
I20240503 14:04:17.626922 15657 leader_election.cc:288] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:17.627255 15660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:17.628549 15660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [term 1 LEADER]: Becoming Leader. State: Replica: 47a15a0b83274f2e96245743094d3bd0, State: Running, Role: LEADER
I20240503 14:04:17.629247 15660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a15a0b83274f2e96245743094d3bd0" member_type: VOTER }
I20240503 14:04:17.629755 15657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:04:17.632156 15662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47a15a0b83274f2e96245743094d3bd0. Latest consensus state: current_term: 1 leader_uuid: "47a15a0b83274f2e96245743094d3bd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a15a0b83274f2e96245743094d3bd0" member_type: VOTER } }
I20240503 14:04:17.632113 15661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47a15a0b83274f2e96245743094d3bd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a15a0b83274f2e96245743094d3bd0" member_type: VOTER } }
I20240503 14:04:17.632824 15662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:17.633041 15661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:17.640513 15665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:04:17.645226 15665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:04:17.646193 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:04:17.654917 15665 catalog_manager.cc:1347] Generated new cluster ID: 78fc7075ce4c440a81c61333eccbdd6d
I20240503 14:04:17.655234 15665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:04:17.669689 15665 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:04:17.671025 15665 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:04:17.699326 15665 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0: Generated new TSK 0
I20240503 14:04:17.700006 15665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:04:17.712831 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:17.719350 15678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:17.720937 15679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:17.721714 15681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:17.722579 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:17.723325 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:17.723520 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:17.723668 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745057723651 us; error 0 us; skew 500 ppm
I20240503 14:04:17.724192 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:17.726738 13945 webserver.cc:460] Webserver started at http://127.13.158.65:36967/ using document root <none> and password file <none>
I20240503 14:04:17.727195 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:17.727376 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:17.727631 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:17.728734 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-0-root/instance:
uuid: "31741fb1bb3245bb928f6902dac6918e"
format_stamp: "Formatted at 2024-05-03 14:04:17 on dist-test-slave-jm3r"
I20240503 14:04:17.733667 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240503 14:04:17.736996 15686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:17.737767 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:04:17.738032 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-0-root
uuid: "31741fb1bb3245bb928f6902dac6918e"
format_stamp: "Formatted at 2024-05-03 14:04:17 on dist-test-slave-jm3r"
I20240503 14:04:17.738315 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:17.771878 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:17.773039 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:17.774485 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:17.776902 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:17.777098 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:17.777324 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:17.777485 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:17.816514 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.65:34809
I20240503 14:04:17.816582 15748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.65:34809 every 8 connection(s)
I20240503 14:04:17.821847 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:17.830149 15753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:17.832139 15754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:17.835865 15749 heartbeater.cc:347] Connected to a master server at 127.13.158.126:46369
I20240503 14:04:17.836283 15749 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:17.837044 15749 heartbeater.cc:511] Master 127.13.158.126:46369 requested a full tablet report, sending...
W20240503 14:04:17.838395 15756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:17.839272 15622 ts_manager.cc:194] Registered new tserver with Master: 31741fb1bb3245bb928f6902dac6918e (127.13.158.65:34809)
I20240503 14:04:17.839278 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:17.840399 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:17.840606 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:17.840775 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745057840756 us; error 0 us; skew 500 ppm
I20240503 14:04:17.841318 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:17.841238 15622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38540
I20240503 14:04:17.844358 13945 webserver.cc:460] Webserver started at http://127.13.158.66:45747/ using document root <none> and password file <none>
I20240503 14:04:17.844892 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:17.845093 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:17.845336 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:17.846560 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-1-root/instance:
uuid: "5183157328e44727b4c934c9c7ff765a"
format_stamp: "Formatted at 2024-05-03 14:04:17 on dist-test-slave-jm3r"
I20240503 14:04:17.851459 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240503 14:04:17.854903 15761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:17.855736 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:04:17.855991 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-1-root
uuid: "5183157328e44727b4c934c9c7ff765a"
format_stamp: "Formatted at 2024-05-03 14:04:17 on dist-test-slave-jm3r"
I20240503 14:04:17.856231 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:17.883687 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:17.884843 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:17.886161 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:17.888422 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:17.888605 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:17.888852 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:17.888993 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:17.926545 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.66:40153
I20240503 14:04:17.926640 15823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.66:40153 every 8 connection(s)
I20240503 14:04:17.931281 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:17.939460 15827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:17.940625 15828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:17.941938 13945 server_base.cc:1034] running on GCE node
W20240503 14:04:17.942551 15830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:17.943918 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:17.944134 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:17.944296 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745057944278 us; error 0 us; skew 500 ppm
I20240503 14:04:17.944797 15824 heartbeater.cc:347] Connected to a master server at 127.13.158.126:46369
I20240503 14:04:17.945135 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:17.945421 15824 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:17.946382 15824 heartbeater.cc:511] Master 127.13.158.126:46369 requested a full tablet report, sending...
I20240503 14:04:17.948096 13945 webserver.cc:460] Webserver started at http://127.13.158.67:42763/ using document root <none> and password file <none>
I20240503 14:04:17.948591 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:17.948638 15622 ts_manager.cc:194] Registered new tserver with Master: 5183157328e44727b4c934c9c7ff765a (127.13.158.66:40153)
I20240503 14:04:17.948818 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:17.949127 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:17.950047 15622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38556
I20240503 14:04:17.950228 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-2-root/instance:
uuid: "0bd9afa20f7848129061f04cdb50d431"
format_stamp: "Formatted at 2024-05-03 14:04:17 on dist-test-slave-jm3r"
I20240503 14:04:17.955106 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 14:04:17.958487 15835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:17.959242 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:04:17.959533 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-2-root
uuid: "0bd9afa20f7848129061f04cdb50d431"
format_stamp: "Formatted at 2024-05-03 14:04:17 on dist-test-slave-jm3r"
I20240503 14:04:17.959805 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:17.969882 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:17.971068 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:17.972470 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:17.974735 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:17.974931 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:17.975155 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:17.975303 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:18.012485 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.67:35459
I20240503 14:04:18.012598 15897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.67:35459 every 8 connection(s)
I20240503 14:04:18.025709 15898 heartbeater.cc:347] Connected to a master server at 127.13.158.126:46369
I20240503 14:04:18.026062 15898 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:18.026836 15898 heartbeater.cc:511] Master 127.13.158.126:46369 requested a full tablet report, sending...
I20240503 14:04:18.028817 15622 ts_manager.cc:194] Registered new tserver with Master: 0bd9afa20f7848129061f04cdb50d431 (127.13.158.67:35459)
I20240503 14:04:18.029423 13945 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014071551s
I20240503 14:04:18.030226 15622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38558
I20240503 14:04:18.844166 15749 heartbeater.cc:503] Master 127.13.158.126:46369 was elected leader, sending a full tablet report...
I20240503 14:04:18.952668 15824 heartbeater.cc:503] Master 127.13.158.126:46369 was elected leader, sending a full tablet report...
I20240503 14:04:19.032974 15898 heartbeater.cc:503] Master 127.13.158.126:46369 was elected leader, sending a full tablet report...
I20240503 14:04:19.062247 13945 test_util.cc:268] Using random seed: 840970704
I20240503 14:04:19.084583 15622 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39800:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:04:19.086694 15622 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:04:19.150067 15788 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee11535cd7a4015a7ea8498d44aab5e (DEFAULT_TABLE table=test-workload [id=3f3e43bc635a440b930cae8e7ab25925]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 14:04:19.150067 15789 tablet_service.cc:1467] Processing CreateTablet for tablet 72ea0d3526a2405c912b70848da3dbe7 (DEFAULT_TABLE table=test-workload [id=3f3e43bc635a440b930cae8e7ab25925]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 14:04:19.152324 15714 tablet_service.cc:1467] Processing CreateTablet for tablet 72ea0d3526a2405c912b70848da3dbe7 (DEFAULT_TABLE table=test-workload [id=3f3e43bc635a440b930cae8e7ab25925]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 14:04:19.152976 15863 tablet_service.cc:1467] Processing CreateTablet for tablet 72ea0d3526a2405c912b70848da3dbe7 (DEFAULT_TABLE table=test-workload [id=3f3e43bc635a440b930cae8e7ab25925]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 14:04:19.153718 15714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ea0d3526a2405c912b70848da3dbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:19.153944 15862 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee11535cd7a4015a7ea8498d44aab5e (DEFAULT_TABLE table=test-workload [id=3f3e43bc635a440b930cae8e7ab25925]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 14:04:19.154625 15713 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee11535cd7a4015a7ea8498d44aab5e (DEFAULT_TABLE table=test-workload [id=3f3e43bc635a440b930cae8e7ab25925]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 14:04:19.151540 15788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ee11535cd7a4015a7ea8498d44aab5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:19.155805 15713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ee11535cd7a4015a7ea8498d44aab5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:19.154299 15863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ea0d3526a2405c912b70848da3dbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:19.157474 15862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ee11535cd7a4015a7ea8498d44aab5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:19.159716 15789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ea0d3526a2405c912b70848da3dbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:19.190217 15919 tablet_bootstrap.cc:492] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a: Bootstrap starting.
I20240503 14:04:19.191706 15920 tablet_bootstrap.cc:492] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431: Bootstrap starting.
I20240503 14:04:19.195323 15918 tablet_bootstrap.cc:492] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e: Bootstrap starting.
I20240503 14:04:19.198796 15920 tablet_bootstrap.cc:654] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:19.199585 15919 tablet_bootstrap.cc:654] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:19.200002 15918 tablet_bootstrap.cc:654] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:19.204396 15920 tablet_bootstrap.cc:492] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431: No bootstrap required, opened a new log
I20240503 14:04:19.204908 15920 ts_tablet_manager.cc:1367] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240503 14:04:19.205010 15918 tablet_bootstrap.cc:492] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e: No bootstrap required, opened a new log
I20240503 14:04:19.205472 15918 ts_tablet_manager.cc:1367] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 14:04:19.205627 15919 tablet_bootstrap.cc:492] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a: No bootstrap required, opened a new log
I20240503 14:04:19.206051 15919 ts_tablet_manager.cc:1367] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20240503 14:04:19.207576 15920 raft_consensus.cc:348] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.208021 15918 raft_consensus.cc:348] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.208421 15920 raft_consensus.cc:374] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:19.208771 15918 raft_consensus.cc:374] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:19.208405 15919 raft_consensus.cc:348] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.208956 15920 raft_consensus.cc:729] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd9afa20f7848129061f04cdb50d431, State: Initialized, Role: FOLLOWER
I20240503 14:04:19.209141 15918 raft_consensus.cc:729] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31741fb1bb3245bb928f6902dac6918e, State: Initialized, Role: FOLLOWER
I20240503 14:04:19.209311 15919 raft_consensus.cc:374] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:19.209694 15919 raft_consensus.cc:729] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5183157328e44727b4c934c9c7ff765a, State: Initialized, Role: FOLLOWER
I20240503 14:04:19.209852 15920 consensus_queue.cc:260] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.209988 15918 consensus_queue.cc:260] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.211612 15919 consensus_queue.cc:260] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.213568 15918 ts_tablet_manager.cc:1398] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240503 14:04:19.214608 15918 tablet_bootstrap.cc:492] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e: Bootstrap starting.
I20240503 14:04:19.219154 15918 tablet_bootstrap.cc:654] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:19.222177 15919 ts_tablet_manager.cc:1398] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a: Time spent starting tablet: real 0.016s user 0.006s sys 0.005s
I20240503 14:04:19.223395 15919 tablet_bootstrap.cc:492] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a: Bootstrap starting.
I20240503 14:04:19.232213 15919 tablet_bootstrap.cc:654] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:19.235687 15920 ts_tablet_manager.cc:1398] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431: Time spent starting tablet: real 0.030s user 0.007s sys 0.009s
I20240503 14:04:19.236877 15920 tablet_bootstrap.cc:492] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431: Bootstrap starting.
I20240503 14:04:19.243670 15919 tablet_bootstrap.cc:492] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a: No bootstrap required, opened a new log
I20240503 14:04:19.243855 15920 tablet_bootstrap.cc:654] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:19.244256 15919 ts_tablet_manager.cc:1367] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.001s
I20240503 14:04:19.246749 15919 raft_consensus.cc:348] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.247298 15919 raft_consensus.cc:374] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:19.247491 15918 tablet_bootstrap.cc:492] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e: No bootstrap required, opened a new log
I20240503 14:04:19.247637 15919 raft_consensus.cc:729] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5183157328e44727b4c934c9c7ff765a, State: Initialized, Role: FOLLOWER
I20240503 14:04:19.248065 15918 ts_tablet_manager.cc:1367] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.012s
I20240503 14:04:19.248471 15919 consensus_queue.cc:260] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.250056 15920 tablet_bootstrap.cc:492] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431: No bootstrap required, opened a new log
I20240503 14:04:19.250468 15919 ts_tablet_manager.cc:1398] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:04:19.250614 15920 ts_tablet_manager.cc:1367] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.002s
I20240503 14:04:19.250697 15918 raft_consensus.cc:348] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.251564 15918 raft_consensus.cc:374] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:19.251880 15918 raft_consensus.cc:729] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31741fb1bb3245bb928f6902dac6918e, State: Initialized, Role: FOLLOWER
I20240503 14:04:19.252831 15925 raft_consensus.cc:482] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:19.252579 15918 consensus_queue.cc:260] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.253311 15925 raft_consensus.cc:504] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.255664 15918 ts_tablet_manager.cc:1398] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:04:19.257313 15920 raft_consensus.cc:348] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.258095 15920 raft_consensus.cc:374] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:19.258417 15920 raft_consensus.cc:729] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd9afa20f7848129061f04cdb50d431, State: Initialized, Role: FOLLOWER
I20240503 14:04:19.258509 15925 leader_election.cc:288] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5183157328e44727b4c934c9c7ff765a (127.13.158.66:40153), 0bd9afa20f7848129061f04cdb50d431 (127.13.158.67:35459)
I20240503 14:04:19.259584 15926 raft_consensus.cc:482] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:19.259285 15920 consensus_queue.cc:260] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.260110 15926 raft_consensus.cc:504] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.261981 15920 ts_tablet_manager.cc:1398] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240503 14:04:19.274554 15926 leader_election.cc:288] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31741fb1bb3245bb928f6902dac6918e (127.13.158.65:34809), 0bd9afa20f7848129061f04cdb50d431 (127.13.158.67:35459)
I20240503 14:04:19.305753 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ea0d3526a2405c912b70848da3dbe7" candidate_uuid: "31741fb1bb3245bb928f6902dac6918e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd9afa20f7848129061f04cdb50d431" is_pre_election: true
I20240503 14:04:19.305779 15799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ea0d3526a2405c912b70848da3dbe7" candidate_uuid: "31741fb1bb3245bb928f6902dac6918e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5183157328e44727b4c934c9c7ff765a" is_pre_election: true
I20240503 14:04:19.306707 15799 raft_consensus.cc:2454] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31741fb1bb3245bb928f6902dac6918e in term 0.
I20240503 14:04:19.306655 15873 raft_consensus.cc:2454] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31741fb1bb3245bb928f6902dac6918e in term 0.
I20240503 14:04:19.308018 15688 leader_election.cc:302] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31741fb1bb3245bb928f6902dac6918e, 5183157328e44727b4c934c9c7ff765a; no voters:
I20240503 14:04:19.308194 15872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ea0d3526a2405c912b70848da3dbe7" candidate_uuid: "5183157328e44727b4c934c9c7ff765a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd9afa20f7848129061f04cdb50d431" is_pre_election: true
I20240503 14:04:19.308851 15925 raft_consensus.cc:2781] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:19.308907 15872 raft_consensus.cc:2454] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5183157328e44727b4c934c9c7ff765a in term 0.
I20240503 14:04:19.309394 15925 raft_consensus.cc:482] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:19.309814 15925 raft_consensus.cc:3037] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:19.310210 15765 leader_election.cc:302] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bd9afa20f7848129061f04cdb50d431, 5183157328e44727b4c934c9c7ff765a; no voters:
I20240503 14:04:19.311049 15926 raft_consensus.cc:2781] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:19.310976 15724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ea0d3526a2405c912b70848da3dbe7" candidate_uuid: "5183157328e44727b4c934c9c7ff765a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31741fb1bb3245bb928f6902dac6918e" is_pre_election: true
I20240503 14:04:19.311501 15926 raft_consensus.cc:482] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:19.311846 15926 raft_consensus.cc:3037] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:19.315582 15925 raft_consensus.cc:504] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.316933 15724 raft_consensus.cc:2379] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5183157328e44727b4c934c9c7ff765a in current term 1: Already voted for candidate 31741fb1bb3245bb928f6902dac6918e in this term.
I20240503 14:04:19.316749 15926 raft_consensus.cc:504] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.317366 15936 raft_consensus.cc:482] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:19.317823 15936 raft_consensus.cc:504] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.318850 15925 leader_election.cc:288] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [CANDIDATE]: Term 1 election: Requested vote from peers 5183157328e44727b4c934c9c7ff765a (127.13.158.66:40153), 0bd9afa20f7848129061f04cdb50d431 (127.13.158.67:35459)
I20240503 14:04:19.319869 15723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ea0d3526a2405c912b70848da3dbe7" candidate_uuid: "5183157328e44727b4c934c9c7ff765a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31741fb1bb3245bb928f6902dac6918e"
I20240503 14:04:19.320391 15799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ea0d3526a2405c912b70848da3dbe7" candidate_uuid: "31741fb1bb3245bb928f6902dac6918e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5183157328e44727b4c934c9c7ff765a"
I20240503 14:04:19.320808 15936 leader_election.cc:288] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31741fb1bb3245bb928f6902dac6918e (127.13.158.65:34809), 0bd9afa20f7848129061f04cdb50d431 (127.13.158.67:35459)
I20240503 14:04:19.321110 15799 raft_consensus.cc:2379] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 31741fb1bb3245bb928f6902dac6918e in current term 1: Already voted for candidate 5183157328e44727b4c934c9c7ff765a in this term.
I20240503 14:04:19.320976 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee11535cd7a4015a7ea8498d44aab5e" candidate_uuid: "5183157328e44727b4c934c9c7ff765a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd9afa20f7848129061f04cdb50d431" is_pre_election: true
I20240503 14:04:19.321635 15873 raft_consensus.cc:2454] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5183157328e44727b4c934c9c7ff765a in term 0.
I20240503 14:04:19.321784 15872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ea0d3526a2405c912b70848da3dbe7" candidate_uuid: "31741fb1bb3245bb928f6902dac6918e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd9afa20f7848129061f04cdb50d431"
I20240503 14:04:19.322374 15872 raft_consensus.cc:3037] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:19.322772 15724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee11535cd7a4015a7ea8498d44aab5e" candidate_uuid: "5183157328e44727b4c934c9c7ff765a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31741fb1bb3245bb928f6902dac6918e" is_pre_election: true
I20240503 14:04:19.322870 15765 leader_election.cc:302] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bd9afa20f7848129061f04cdb50d431, 5183157328e44727b4c934c9c7ff765a; no voters:
I20240503 14:04:19.323415 15724 raft_consensus.cc:2454] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5183157328e44727b4c934c9c7ff765a in term 0.
I20240503 14:04:19.320700 15723 raft_consensus.cc:2379] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5183157328e44727b4c934c9c7ff765a in current term 1: Already voted for candidate 31741fb1bb3245bb928f6902dac6918e in this term.
I20240503 14:04:19.324158 15926 leader_election.cc:288] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [CANDIDATE]: Term 1 election: Requested vote from peers 31741fb1bb3245bb928f6902dac6918e (127.13.158.65:34809), 0bd9afa20f7848129061f04cdb50d431 (127.13.158.67:35459)
I20240503 14:04:19.324695 15926 raft_consensus.cc:2781] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:19.325039 15926 raft_consensus.cc:482] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:19.325505 15926 raft_consensus.cc:3037] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:19.325366 15873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ea0d3526a2405c912b70848da3dbe7" candidate_uuid: "5183157328e44727b4c934c9c7ff765a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd9afa20f7848129061f04cdb50d431"
I20240503 14:04:19.327054 15765 leader_election.cc:302] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5183157328e44727b4c934c9c7ff765a; no voters: 0bd9afa20f7848129061f04cdb50d431, 31741fb1bb3245bb928f6902dac6918e
I20240503 14:04:19.327881 15936 raft_consensus.cc:2726] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:04:19.328780 15872 raft_consensus.cc:2454] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31741fb1bb3245bb928f6902dac6918e in term 1.
I20240503 14:04:19.329790 15690 leader_election.cc:302] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bd9afa20f7848129061f04cdb50d431, 31741fb1bb3245bb928f6902dac6918e; no voters: 5183157328e44727b4c934c9c7ff765a
I20240503 14:04:19.330603 15925 raft_consensus.cc:2781] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:19.331115 15925 raft_consensus.cc:686] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 1 LEADER]: Becoming Leader. State: Replica: 31741fb1bb3245bb928f6902dac6918e, State: Running, Role: LEADER
I20240503 14:04:19.331660 15926 raft_consensus.cc:504] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.333657 15926 leader_election.cc:288] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [CANDIDATE]: Term 1 election: Requested vote from peers 31741fb1bb3245bb928f6902dac6918e (127.13.158.65:34809), 0bd9afa20f7848129061f04cdb50d431 (127.13.158.67:35459)
I20240503 14:04:19.332594 15925 consensus_queue.cc:237] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.334506 15724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee11535cd7a4015a7ea8498d44aab5e" candidate_uuid: "5183157328e44727b4c934c9c7ff765a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31741fb1bb3245bb928f6902dac6918e"
I20240503 14:04:19.334610 15872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee11535cd7a4015a7ea8498d44aab5e" candidate_uuid: "5183157328e44727b4c934c9c7ff765a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd9afa20f7848129061f04cdb50d431"
I20240503 14:04:19.335106 15724 raft_consensus.cc:3037] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:19.335381 15872 raft_consensus.cc:3037] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:19.341159 15872 raft_consensus.cc:2454] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5183157328e44727b4c934c9c7ff765a in term 1.
I20240503 14:04:19.342114 15765 leader_election.cc:302] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bd9afa20f7848129061f04cdb50d431, 5183157328e44727b4c934c9c7ff765a; no voters:
I20240503 14:04:19.342872 15926 raft_consensus.cc:2781] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:19.342927 15724 raft_consensus.cc:2454] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5183157328e44727b4c934c9c7ff765a in term 1.
I20240503 14:04:19.343343 15926 raft_consensus.cc:686] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 1 LEADER]: Becoming Leader. State: Replica: 5183157328e44727b4c934c9c7ff765a, State: Running, Role: LEADER
I20240503 14:04:19.344483 15926 consensus_queue.cc:237] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } }
I20240503 14:04:19.345197 15622 catalog_manager.cc:5484] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e reported cstate change: term changed from 0 to 1, leader changed from <none> to 31741fb1bb3245bb928f6902dac6918e (127.13.158.65). New cstate: current_term: 1 leader_uuid: "31741fb1bb3245bb928f6902dac6918e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:19.352186 15620 catalog_manager.cc:5484] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5183157328e44727b4c934c9c7ff765a (127.13.158.66). New cstate: current_term: 1 leader_uuid: "5183157328e44727b4c934c9c7ff765a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:19.644554 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:19.646456 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:19.646790 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:19.774899 15926 consensus_queue.cc:1035] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:19.785822 15926 consensus_queue.cc:1035] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:19.839852 15925 consensus_queue.cc:1035] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:19.851310 15946 consensus_queue.cc:1035] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:21.647681 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:21.649786 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:21.650146 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:23.651036 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:23.653146 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:23.653542 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:25.654486 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:25.656315 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:25.656656 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:27.657806 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:27.660058 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:27.660389 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:29.661206 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:29.663334 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:29.663681 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:30.448136 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:30.455868 15955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:30.456426 15956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:30.458408 15958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:30.460289 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:30.461366 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:30.461578 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:30.461735 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745070461714 us; error 0 us; skew 500 ppm
I20240503 14:04:30.462291 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:30.465013 13945 webserver.cc:460] Webserver started at http://127.13.158.68:41229/ using document root <none> and password file <none>
I20240503 14:04:30.465481 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:30.465659 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:30.465919 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:30.467307 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-3-root/instance:
uuid: "9f31aeda6c6d45ca9da8453b0836a6fa"
format_stamp: "Formatted at 2024-05-03 14:04:30 on dist-test-slave-jm3r"
I20240503 14:04:30.472065 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240503 14:04:30.476248 15963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:30.477316 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:04:30.477664 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-3-root
uuid: "9f31aeda6c6d45ca9da8453b0836a6fa"
format_stamp: "Formatted at 2024-05-03 14:04:30 on dist-test-slave-jm3r"
I20240503 14:04:30.477960 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:30.494500 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:30.495759 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:30.497277 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:30.499780 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:30.500003 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:30.500231 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:30.500393 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:30.542946 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.68:44003
I20240503 14:04:30.543025 16025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.68:44003 every 8 connection(s)
I20240503 14:04:30.557261 16026 heartbeater.cc:347] Connected to a master server at 127.13.158.126:46369
I20240503 14:04:30.557686 16026 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:30.558384 16026 heartbeater.cc:511] Master 127.13.158.126:46369 requested a full tablet report, sending...
I20240503 14:04:30.560263 15620 ts_manager.cc:194] Registered new tserver with Master: 9f31aeda6c6d45ca9da8453b0836a6fa (127.13.158.68:44003)
I20240503 14:04:30.561866 15620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45156
I20240503 14:04:30.703572 15724 consensus_queue.cc:237] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: true } }
I20240503 14:04:30.713008 15799 raft_consensus.cc:1261] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Refusing update from remote peer 31741fb1bb3245bb928f6902dac6918e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:04:30.714370 15872 raft_consensus.cc:1261] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 1 FOLLOWER]: Refusing update from remote peer 31741fb1bb3245bb928f6902dac6918e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:04:30.714692 16033 consensus_queue.cc:1035] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:04:30.715942 16034 consensus_queue.cc:1035] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:04:30.726284 15690 consensus_peers.cc:485] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e -> Peer 9f31aeda6c6d45ca9da8453b0836a6fa (127.13.158.68:44003): Couldn't send request to peer 9f31aeda6c6d45ca9da8453b0836a6fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ea0d3526a2405c912b70848da3dbe7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:30.728078 16034 raft_consensus.cc:2932] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9f31aeda6c6d45ca9da8453b0836a6fa (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: true } } }
I20240503 14:04:30.730193 15872 raft_consensus.cc:2932] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9f31aeda6c6d45ca9da8453b0836a6fa (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: true } } }
I20240503 14:04:30.736786 15799 raft_consensus.cc:2932] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9f31aeda6c6d45ca9da8453b0836a6fa (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: true } } }
I20240503 14:04:30.746618 15620 catalog_manager.cc:5484] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e reported cstate change: config changed from index -1 to 2, NON_VOTER 9f31aeda6c6d45ca9da8453b0836a6fa (127.13.158.68) added. New cstate: current_term: 1 leader_uuid: "31741fb1bb3245bb928f6902dac6918e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:31.242471 16044 ts_tablet_manager.cc:902] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa: Initiating tablet copy from peer 31741fb1bb3245bb928f6902dac6918e (127.13.158.65:34809)
I20240503 14:04:31.246033 16044 tablet_copy_client.cc:308] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa: tablet copy: Beginning tablet copy session from remote peer at address 127.13.158.65:34809
I20240503 14:04:31.257869 15734 tablet_copy_service.cc:140] P 31741fb1bb3245bb928f6902dac6918e: Received BeginTabletCopySession request for tablet 72ea0d3526a2405c912b70848da3dbe7 from peer 9f31aeda6c6d45ca9da8453b0836a6fa ({username='slave'} at 127.0.0.1:59428)
I20240503 14:04:31.258505 15734 tablet_copy_service.cc:161] P 31741fb1bb3245bb928f6902dac6918e: Beginning new tablet copy session on tablet 72ea0d3526a2405c912b70848da3dbe7 from peer 9f31aeda6c6d45ca9da8453b0836a6fa at {username='slave'} at 127.0.0.1:59428: session id = 9f31aeda6c6d45ca9da8453b0836a6fa-72ea0d3526a2405c912b70848da3dbe7
I20240503 14:04:31.265061 15734 tablet_copy_source_session.cc:205] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:04:31.267737 16044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ea0d3526a2405c912b70848da3dbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:31.282210 16044 tablet_copy_client.cc:783] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa: tablet copy: Starting download of 0 data blocks...
I20240503 14:04:31.282899 16044 tablet_copy_client.cc:647] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa: tablet copy: Starting download of 1 WAL segments...
I20240503 14:04:31.286686 16044 tablet_copy_client.cc:523] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:04:31.293679 16044 tablet_bootstrap.cc:492] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa: Bootstrap starting.
I20240503 14:04:31.310534 16044 tablet_bootstrap.cc:492] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:04:31.311319 16044 tablet_bootstrap.cc:492] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa: Bootstrap complete.
I20240503 14:04:31.312007 16044 ts_tablet_manager.cc:1367] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.001s
I20240503 14:04:31.314399 16044 raft_consensus.cc:348] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: true } }
I20240503 14:04:31.315300 16044 raft_consensus.cc:729] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f31aeda6c6d45ca9da8453b0836a6fa, State: Initialized, Role: LEARNER
I20240503 14:04:31.315965 16044 consensus_queue.cc:260] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: true } }
I20240503 14:04:31.318028 16026 heartbeater.cc:503] Master 127.13.158.126:46369 was elected leader, sending a full tablet report...
I20240503 14:04:31.318395 16044 ts_tablet_manager.cc:1398] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20240503 14:04:31.320067 15734 tablet_copy_service.cc:342] P 31741fb1bb3245bb928f6902dac6918e: Request end of tablet copy session 9f31aeda6c6d45ca9da8453b0836a6fa-72ea0d3526a2405c912b70848da3dbe7 received from {username='slave'} at 127.0.0.1:59428
I20240503 14:04:31.320422 15734 tablet_copy_service.cc:433] P 31741fb1bb3245bb928f6902dac6918e: ending tablet copy session 9f31aeda6c6d45ca9da8453b0836a6fa-72ea0d3526a2405c912b70848da3dbe7 on tablet 72ea0d3526a2405c912b70848da3dbe7 with peer 9f31aeda6c6d45ca9da8453b0836a6fa
I20240503 14:04:31.644447 16001 raft_consensus.cc:1203] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:04:31.664615 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:31.666924 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:31.667462 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:32.229694 16054 raft_consensus.cc:1050] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e: attempting to promote NON_VOTER 9f31aeda6c6d45ca9da8453b0836a6fa to VOTER
I20240503 14:04:32.232017 16054 consensus_queue.cc:237] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false } }
I20240503 14:04:32.238611 16001 raft_consensus.cc:1261] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa [term 1 LEARNER]: Refusing update from remote peer 31741fb1bb3245bb928f6902dac6918e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:04:32.240141 15872 raft_consensus.cc:1261] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 1 FOLLOWER]: Refusing update from remote peer 31741fb1bb3245bb928f6902dac6918e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:04:32.240826 16055 consensus_queue.cc:1035] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:32.242372 16057 consensus_queue.cc:1035] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:32.245330 15799 raft_consensus.cc:1261] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Refusing update from remote peer 31741fb1bb3245bb928f6902dac6918e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:04:32.246855 16057 consensus_queue.cc:1035] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:32.259385 16057 raft_consensus.cc:2932] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9f31aeda6c6d45ca9da8453b0836a6fa (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false } } }
I20240503 14:04:32.261512 16001 raft_consensus.cc:2932] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9f31aeda6c6d45ca9da8453b0836a6fa (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false } } }
I20240503 14:04:32.263967 15872 raft_consensus.cc:2932] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9f31aeda6c6d45ca9da8453b0836a6fa (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false } } }
I20240503 14:04:32.271757 15799 raft_consensus.cc:2932] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9f31aeda6c6d45ca9da8453b0836a6fa (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false } } }
I20240503 14:04:32.281412 15622 catalog_manager.cc:5484] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e reported cstate change: config changed from index 2 to 3, 9f31aeda6c6d45ca9da8453b0836a6fa (127.13.158.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31741fb1bb3245bb928f6902dac6918e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bd9afa20f7848129061f04cdb50d431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:04:32.307786 15723 consensus_queue.cc:237] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false } }
I20240503 14:04:32.312466 16001 raft_consensus.cc:1261] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa [term 1 FOLLOWER]: Refusing update from remote peer 31741fb1bb3245bb928f6902dac6918e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:04:32.313086 15799 raft_consensus.cc:1261] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Refusing update from remote peer 31741fb1bb3245bb928f6902dac6918e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:04:32.313725 16057 consensus_queue.cc:1035] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:32.315521 16054 consensus_queue.cc:1035] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:32.320132 16057 raft_consensus.cc:2932] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0bd9afa20f7848129061f04cdb50d431 (127.13.158.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false } } }
I20240503 14:04:32.325265 15799 raft_consensus.cc:2932] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0bd9afa20f7848129061f04cdb50d431 (127.13.158.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false } } }
I20240503 14:04:32.321986 16001 raft_consensus.cc:2932] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0bd9afa20f7848129061f04cdb50d431 (127.13.158.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false } } }
I20240503 14:04:32.329643 15608 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72ea0d3526a2405c912b70848da3dbe7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:04:32.335263 15619 catalog_manager.cc:5484] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e reported cstate change: config changed from index 3 to 4, VOTER 0bd9afa20f7848129061f04cdb50d431 (127.13.158.67) evicted. New cstate: current_term: 1 leader_uuid: "31741fb1bb3245bb928f6902dac6918e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5183157328e44727b4c934c9c7ff765a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 40153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31741fb1bb3245bb928f6902dac6918e" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f31aeda6c6d45ca9da8453b0836a6fa" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 44003 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:04:32.347069 15863 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ea0d3526a2405c912b70848da3dbe7 with delete_type TABLET_DATA_TOMBSTONED (TS 0bd9afa20f7848129061f04cdb50d431 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54692
I20240503 14:04:32.349126 16067 tablet_replica.cc:330] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431: stopping tablet replica
I20240503 14:04:32.349869 16067 raft_consensus.cc:2229] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:32.350540 16067 raft_consensus.cc:2258] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:32.353382 16067 ts_tablet_manager.cc:1875] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:04:32.366019 16067 ts_tablet_manager.cc:1888] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:04:32.366348 16067 log.cc:1196] T 72ea0d3526a2405c912b70848da3dbe7 P 0bd9afa20f7848129061f04cdb50d431: Deleting WAL directory at /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714745036514908-13945-0/minicluster-data/ts-2-root/wals/72ea0d3526a2405c912b70848da3dbe7
I20240503 14:04:32.367905 15609 catalog_manager.cc:4830] TS 0bd9afa20f7848129061f04cdb50d431 (127.13.158.67:35459): tablet 72ea0d3526a2405c912b70848da3dbe7 (table test-workload [id=3f3e43bc635a440b930cae8e7ab25925]) successfully deleted
I20240503 14:04:33.668356 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:33.670369 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:33.670753 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:35.671804 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:35.673969 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:35.674337 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 14:04:37.363888 15745 debug-util.cc:398] Leaking SignalData structure 0x7b08000cb2e0 after lost signal to thread 13948
W20240503 14:04:37.364924 15745 debug-util.cc:398] Leaking SignalData structure 0x7b08000816a0 after lost signal to thread 15656
W20240503 14:04:37.365942 15745 debug-util.cc:398] Leaking SignalData structure 0x7b0800230de0 after lost signal to thread 15748
W20240503 14:04:37.366978 15745 debug-util.cc:398] Leaking SignalData structure 0x7b08001645e0 after lost signal to thread 15823
W20240503 14:04:37.368009 15745 debug-util.cc:398] Leaking SignalData structure 0x7b0800167ae0 after lost signal to thread 15897
W20240503 14:04:37.369002 15745 debug-util.cc:398] Leaking SignalData structure 0x7b0800169e40 after lost signal to thread 16025
I20240503 14:04:37.675374 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:37.677309 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:37.677642 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:39.678628 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:39.680534 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:39.680864 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:41.681752 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:41.683652 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:41.683996 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:43.684888 15675 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:43.686928 15675 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:43.687297 15675 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:44.570952 13945 tablet_server.cc:178] TabletServer@127.13.158.65:0 shutting down...
I20240503 14:04:44.595322 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:44.596047 13945 tablet_replica.cc:330] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e: stopping tablet replica
I20240503 14:04:44.596676 13945 raft_consensus.cc:2229] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:44.597218 13945 raft_consensus.cc:2258] T 8ee11535cd7a4015a7ea8498d44aab5e P 31741fb1bb3245bb928f6902dac6918e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:44.599421 13945 tablet_replica.cc:330] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e: stopping tablet replica
I20240503 14:04:44.599884 13945 raft_consensus.cc:2229] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:44.600776 13945 raft_consensus.cc:2258] T 72ea0d3526a2405c912b70848da3dbe7 P 31741fb1bb3245bb928f6902dac6918e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:44.644065 13945 tablet_server.cc:194] TabletServer@127.13.158.65:0 shutdown complete.
I20240503 14:04:44.656877 13945 tablet_server.cc:178] TabletServer@127.13.158.66:0 shutting down...
I20240503 14:04:44.679174 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:44.679818 13945 tablet_replica.cc:330] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a: stopping tablet replica
I20240503 14:04:44.680404 13945 raft_consensus.cc:2229] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:44.681123 13945 raft_consensus.cc:2258] T 72ea0d3526a2405c912b70848da3dbe7 P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:44.683092 13945 tablet_replica.cc:330] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a: stopping tablet replica
I20240503 14:04:44.683586 13945 raft_consensus.cc:2229] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:44.684379 13945 raft_consensus.cc:2258] T 8ee11535cd7a4015a7ea8498d44aab5e P 5183157328e44727b4c934c9c7ff765a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:44.704586 13945 tablet_server.cc:194] TabletServer@127.13.158.66:0 shutdown complete.
I20240503 14:04:44.715972 13945 tablet_server.cc:178] TabletServer@127.13.158.67:0 shutting down...
I20240503 14:04:44.739275 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:44.739971 13945 tablet_replica.cc:330] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431: stopping tablet replica
I20240503 14:04:44.740550 13945 raft_consensus.cc:2229] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:44.740988 13945 raft_consensus.cc:2258] T 8ee11535cd7a4015a7ea8498d44aab5e P 0bd9afa20f7848129061f04cdb50d431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:44.762032 13945 tablet_server.cc:194] TabletServer@127.13.158.67:0 shutdown complete.
I20240503 14:04:44.773545 13945 tablet_server.cc:178] TabletServer@127.13.158.68:0 shutting down...
I20240503 14:04:44.794225 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:44.794910 13945 tablet_replica.cc:330] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa: stopping tablet replica
I20240503 14:04:44.795478 13945 raft_consensus.cc:2229] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:44.796201 13945 raft_consensus.cc:2258] T 72ea0d3526a2405c912b70848da3dbe7 P 9f31aeda6c6d45ca9da8453b0836a6fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:44.815204 13945 tablet_server.cc:194] TabletServer@127.13.158.68:0 shutdown complete.
I20240503 14:04:44.824987 13945 master.cc:537] Master@127.13.158.126:46369 shutting down...
I20240503 14:04:44.844079 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:44.844660 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:44.844957 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 47a15a0b83274f2e96245743094d3bd0: stopping tablet replica
I20240503 14:04:44.862777 13945 master.cc:552] Master@127.13.158.126:46369 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (27400 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240503 14:04:44.898321 13945 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.13.158.126:40671
I20240503 14:04:44.899397 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:44.905623 16077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:44.906288 16078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:44.906960 16080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:44.907590 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:44.908490 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:44.908686 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:44.908828 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745084908815 us; error 0 us; skew 500 ppm
I20240503 14:04:44.909325 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:44.917227 13945 webserver.cc:460] Webserver started at http://127.13.158.126:43179/ using document root <none> and password file <none>
I20240503 14:04:44.917757 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:44.917956 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:44.918241 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:44.919663 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714745036514908-13945-0/minicluster-data/master-0-root/instance:
uuid: "9b66cb21ed324d7a85fc3d48438d7d00"
format_stamp: "Formatted at 2024-05-03 14:04:44 on dist-test-slave-jm3r"
I20240503 14:04:44.924362 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:04:44.927882 16085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:44.928647 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:04:44.928966 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714745036514908-13945-0/minicluster-data/master-0-root
uuid: "9b66cb21ed324d7a85fc3d48438d7d00"
format_stamp: "Formatted at 2024-05-03 14:04:44 on dist-test-slave-jm3r"
I20240503 14:04:44.929265 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714745036514908-13945-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714745036514908-13945-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714745036514908-13945-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:44.948094 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:44.949345 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:44.988483 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.126:40671
I20240503 14:04:44.988577 16136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.126:40671 every 8 connection(s)
I20240503 14:04:44.992369 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:45.002952 16137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00: Bootstrap starting.
I20240503 14:04:45.007231 16137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:45.011576 16137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00: No bootstrap required, opened a new log
I20240503 14:04:45.013614 16137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b66cb21ed324d7a85fc3d48438d7d00" member_type: VOTER }
I20240503 14:04:45.014123 16137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:45.014314 16137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b66cb21ed324d7a85fc3d48438d7d00, State: Initialized, Role: FOLLOWER
I20240503 14:04:45.014907 16137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b66cb21ed324d7a85fc3d48438d7d00" member_type: VOTER }
I20240503 14:04:45.015375 16137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:45.015563 16137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:45.015781 16137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:45.020335 16137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b66cb21ed324d7a85fc3d48438d7d00" member_type: VOTER }
I20240503 14:04:45.020854 16137 leader_election.cc:302] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b66cb21ed324d7a85fc3d48438d7d00; no voters:
I20240503 14:04:45.021920 16137 leader_election.cc:288] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:45.022318 16140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:45.024029 16140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [term 1 LEADER]: Becoming Leader. State: Replica: 9b66cb21ed324d7a85fc3d48438d7d00, State: Running, Role: LEADER
I20240503 14:04:45.025997 16140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b66cb21ed324d7a85fc3d48438d7d00" member_type: VOTER }
I20240503 14:04:45.026683 16137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:04:45.038372 16141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b66cb21ed324d7a85fc3d48438d7d00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b66cb21ed324d7a85fc3d48438d7d00" member_type: VOTER } }
I20240503 14:04:45.038321 16142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b66cb21ed324d7a85fc3d48438d7d00. Latest consensus state: current_term: 1 leader_uuid: "9b66cb21ed324d7a85fc3d48438d7d00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b66cb21ed324d7a85fc3d48438d7d00" member_type: VOTER } }
I20240503 14:04:45.039307 16142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:45.039211 16141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:45.044554 16149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:04:45.050055 16149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:04:45.051141 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:04:45.058712 16149 catalog_manager.cc:1347] Generated new cluster ID: d4e2d41025e043bcb421bfd925ed7802
I20240503 14:04:45.059072 16149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:04:45.075965 16149 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:04:45.077263 16149 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:04:45.090241 16149 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00: Generated new TSK 0
I20240503 14:04:45.090878 16149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:04:45.116961 13945 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.000212748s
W20240503 14:04:48.726259 16133 debug-util.cc:398] Leaking SignalData structure 0x7b080004b660 after lost signal to thread 13948
W20240503 14:04:48.727218 16133 debug-util.cc:398] Leaking SignalData structure 0x7b080004bda0 after lost signal to thread 16136
I20240503 14:04:57.227864 13945 master.cc:537] Master@127.13.158.126:40671 shutting down...
I20240503 14:04:57.251894 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:57.252698 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:57.253031 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9b66cb21ed324d7a85fc3d48438d7d00: stopping tablet replica
I20240503 14:04:57.287806 13945 master.cc:552] Master@127.13.158.126:40671 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12418 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240503 14:04:57.317204 13945 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.13.158.126:37361
I20240503 14:04:57.318408 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:57.325765 16161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:57.326458 16162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:57.327040 16164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:57.327704 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:57.329546 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:57.329741 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:57.329885 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745097329867 us; error 0 us; skew 500 ppm
I20240503 14:04:57.330390 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:57.337298 13945 webserver.cc:460] Webserver started at http://127.13.158.126:45257/ using document root <none> and password file <none>
I20240503 14:04:57.337803 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:57.337965 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:57.338245 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:57.339586 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/master-0-root/instance:
uuid: "56e235cda8aa48adaa1f97e7a5f0a627"
format_stamp: "Formatted at 2024-05-03 14:04:57 on dist-test-slave-jm3r"
I20240503 14:04:57.344415 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:04:57.347962 16169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:57.348943 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:04:57.349273 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/master-0-root
uuid: "56e235cda8aa48adaa1f97e7a5f0a627"
format_stamp: "Formatted at 2024-05-03 14:04:57 on dist-test-slave-jm3r"
I20240503 14:04:57.349571 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:57.363808 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:57.365001 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:57.402693 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.126:37361
I20240503 14:04:57.402906 16220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.126:37361 every 8 connection(s)
I20240503 14:04:57.406697 16221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.418211 16221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627: Bootstrap starting.
I20240503 14:04:57.422562 16221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:57.426775 16221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627: No bootstrap required, opened a new log
I20240503 14:04:57.428800 16221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e235cda8aa48adaa1f97e7a5f0a627" member_type: VOTER }
I20240503 14:04:57.429279 16221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:57.429519 16221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e235cda8aa48adaa1f97e7a5f0a627, State: Initialized, Role: FOLLOWER
I20240503 14:04:57.430099 16221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e235cda8aa48adaa1f97e7a5f0a627" member_type: VOTER }
I20240503 14:04:57.430563 16221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:57.430771 16221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:57.431007 16221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:57.435448 16221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e235cda8aa48adaa1f97e7a5f0a627" member_type: VOTER }
I20240503 14:04:57.435945 16221 leader_election.cc:302] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56e235cda8aa48adaa1f97e7a5f0a627; no voters:
I20240503 14:04:57.437011 16221 leader_election.cc:288] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:57.437311 16224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:57.438558 16224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [term 1 LEADER]: Becoming Leader. State: Replica: 56e235cda8aa48adaa1f97e7a5f0a627, State: Running, Role: LEADER
I20240503 14:04:57.439242 16224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e235cda8aa48adaa1f97e7a5f0a627" member_type: VOTER }
I20240503 14:04:57.439857 16221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:04:57.444654 16226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56e235cda8aa48adaa1f97e7a5f0a627. Latest consensus state: current_term: 1 leader_uuid: "56e235cda8aa48adaa1f97e7a5f0a627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e235cda8aa48adaa1f97e7a5f0a627" member_type: VOTER } }
I20240503 14:04:57.445027 16225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56e235cda8aa48adaa1f97e7a5f0a627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e235cda8aa48adaa1f97e7a5f0a627" member_type: VOTER } }
I20240503 14:04:57.445314 16226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:57.445544 16225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:57.447126 16232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:04:57.452780 16232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:04:57.453948 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:04:57.461514 16232 catalog_manager.cc:1347] Generated new cluster ID: 3dc015a995f84f1bb686e5ab9985010d
I20240503 14:04:57.461827 16232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:04:57.483687 16232 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:04:57.484974 16232 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:04:57.503518 16232 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627: Generated new TSK 0
I20240503 14:04:57.504142 16232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:04:57.520800 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:57.527698 16242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:57.528409 16243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:57.529889 16245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:57.530478 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:57.531273 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:57.531452 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:57.531606 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745097531590 us; error 0 us; skew 500 ppm
I20240503 14:04:57.532152 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:57.534415 13945 webserver.cc:460] Webserver started at http://127.13.158.65:39597/ using document root <none> and password file <none>
I20240503 14:04:57.534893 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:57.535059 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:57.535306 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:57.536466 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-0-root/instance:
uuid: "160e10e6f10f41fea3a0041aac074b22"
format_stamp: "Formatted at 2024-05-03 14:04:57 on dist-test-slave-jm3r"
I20240503 14:04:57.541278 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:04:57.544639 16250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:57.545383 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:04:57.545675 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-0-root
uuid: "160e10e6f10f41fea3a0041aac074b22"
format_stamp: "Formatted at 2024-05-03 14:04:57 on dist-test-slave-jm3r"
I20240503 14:04:57.545948 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:57.558219 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:57.559592 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:57.561146 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:57.563706 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:57.563916 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:57.564152 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:57.564314 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:57.605122 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.65:40563
I20240503 14:04:57.605237 16312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.65:40563 every 8 connection(s)
I20240503 14:04:57.610132 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:57.620363 16317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:57.620926 16318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:57.626345 13945 server_base.cc:1034] running on GCE node
W20240503 14:04:57.627295 16320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:57.628320 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:57.628567 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:57.628772 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745097628751 us; error 0 us; skew 500 ppm
I20240503 14:04:57.628822 16313 heartbeater.cc:347] Connected to a master server at 127.13.158.126:37361
I20240503 14:04:57.629259 16313 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:57.629611 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:57.630236 16313 heartbeater.cc:511] Master 127.13.158.126:37361 requested a full tablet report, sending...
I20240503 14:04:57.632503 13945 webserver.cc:460] Webserver started at http://127.13.158.66:41575/ using document root <none> and password file <none>
I20240503 14:04:57.632642 16186 ts_manager.cc:194] Registered new tserver with Master: 160e10e6f10f41fea3a0041aac074b22 (127.13.158.65:40563)
I20240503 14:04:57.633064 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:57.633297 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:57.633541 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:57.634615 16186 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33970
I20240503 14:04:57.634799 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-1-root/instance:
uuid: "edf648b686204f3d8e551f6796fdfc56"
format_stamp: "Formatted at 2024-05-03 14:04:57 on dist-test-slave-jm3r"
I20240503 14:04:57.639916 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:04:57.643288 16325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:57.644013 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:04:57.644297 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-1-root
uuid: "edf648b686204f3d8e551f6796fdfc56"
format_stamp: "Formatted at 2024-05-03 14:04:57 on dist-test-slave-jm3r"
I20240503 14:04:57.644563 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:57.657321 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:57.658594 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:57.659935 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:57.662093 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:57.662284 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:57.662544 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:57.662693 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:57.700251 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.66:44945
I20240503 14:04:57.700342 16387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.66:44945 every 8 connection(s)
I20240503 14:04:57.704874 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:57.712296 16391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:57.713747 16392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:57.716127 16394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:57.716588 16388 heartbeater.cc:347] Connected to a master server at 127.13.158.126:37361
I20240503 14:04:57.717022 16388 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:57.717032 13945 server_base.cc:1034] running on GCE node
I20240503 14:04:57.717927 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 14:04:57.717954 16388 heartbeater.cc:511] Master 127.13.158.126:37361 requested a full tablet report, sending...
W20240503 14:04:57.718191 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:57.718410 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745097718393 us; error 0 us; skew 500 ppm
I20240503 14:04:57.719017 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:57.719983 16186 ts_manager.cc:194] Registered new tserver with Master: edf648b686204f3d8e551f6796fdfc56 (127.13.158.66:44945)
I20240503 14:04:57.721575 13945 webserver.cc:460] Webserver started at http://127.13.158.67:34669/ using document root <none> and password file <none>
I20240503 14:04:57.721588 16186 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33982
I20240503 14:04:57.722165 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:57.722375 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:57.722680 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:57.723876 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-2-root/instance:
uuid: "29613535e6814d228e782f39976569a1"
format_stamp: "Formatted at 2024-05-03 14:04:57 on dist-test-slave-jm3r"
I20240503 14:04:57.728210 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:04:57.731379 16399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:57.732218 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:04:57.732467 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-2-root
uuid: "29613535e6814d228e782f39976569a1"
format_stamp: "Formatted at 2024-05-03 14:04:57 on dist-test-slave-jm3r"
I20240503 14:04:57.732720 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714745036514908-13945-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:57.751161 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:57.752375 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:57.753822 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:57.756026 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:57.756212 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:57.756439 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:57.756575 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:57.795711 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.67:35405
I20240503 14:04:57.795794 16461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.67:35405 every 8 connection(s)
I20240503 14:04:57.808516 16462 heartbeater.cc:347] Connected to a master server at 127.13.158.126:37361
I20240503 14:04:57.808900 16462 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:57.809614 16462 heartbeater.cc:511] Master 127.13.158.126:37361 requested a full tablet report, sending...
I20240503 14:04:57.811503 16186 ts_manager.cc:194] Registered new tserver with Master: 29613535e6814d228e782f39976569a1 (127.13.158.67:35405)
I20240503 14:04:57.812274 13945 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013452666s
I20240503 14:04:57.812830 16186 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33994
I20240503 14:04:58.637733 16313 heartbeater.cc:503] Master 127.13.158.126:37361 was elected leader, sending a full tablet report...
I20240503 14:04:58.724074 16388 heartbeater.cc:503] Master 127.13.158.126:37361 was elected leader, sending a full tablet report...
I20240503 14:04:58.815503 16462 heartbeater.cc:503] Master 127.13.158.126:37361 was elected leader, sending a full tablet report...
I20240503 14:04:59.453828 16239 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:01.454777 16239 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:03.455756 16239 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:05.456745 16239 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:07.458132 16239 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:09.459659 16239 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:09.908082 13945 tablet_server.cc:178] TabletServer@127.13.158.65:0 shutting down...
I20240503 14:05:09.932433 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:09.949189 13945 tablet_server.cc:194] TabletServer@127.13.158.65:0 shutdown complete.
I20240503 14:05:09.958369 13945 tablet_server.cc:178] TabletServer@127.13.158.66:0 shutting down...
I20240503 14:05:09.977669 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:09.993295 13945 tablet_server.cc:194] TabletServer@127.13.158.66:0 shutdown complete.
I20240503 14:05:10.002007 13945 tablet_server.cc:178] TabletServer@127.13.158.67:0 shutting down...
I20240503 14:05:10.018832 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:10.034373 13945 tablet_server.cc:194] TabletServer@127.13.158.67:0 shutdown complete.
I20240503 14:05:10.043076 13945 master.cc:537] Master@127.13.158.126:37361 shutting down...
I20240503 14:05:10.059199 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:05:10.059937 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:10.060304 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 56e235cda8aa48adaa1f97e7a5f0a627: stopping tablet replica
I20240503 14:05:10.078022 13945 master.cc:552] Master@127.13.158.126:37361 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12789 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240503 14:05:10.106736 13945 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.13.158.126:40965
I20240503 14:05:10.107817 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:10.113763 16467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:10.114706 16468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:10.115514 16470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:10.116314 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:10.117058 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:10.117254 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:10.117398 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745110117381 us; error 0 us; skew 500 ppm
I20240503 14:05:10.117892 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:10.120543 13945 webserver.cc:460] Webserver started at http://127.13.158.126:36019/ using document root <none> and password file <none>
I20240503 14:05:10.121011 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:10.121183 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:10.121439 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:10.122753 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/master-0-root/instance:
uuid: "d9862abea47943709ee653c229ff753a"
format_stamp: "Formatted at 2024-05-03 14:05:10 on dist-test-slave-jm3r"
I20240503 14:05:10.128232 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20240503 14:05:10.131661 16475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.132476 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:05:10.132768 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/master-0-root
uuid: "d9862abea47943709ee653c229ff753a"
format_stamp: "Formatted at 2024-05-03 14:05:10 on dist-test-slave-jm3r"
I20240503 14:05:10.133051 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:10.162129 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:10.163409 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:10.200363 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.126:40965
I20240503 14:05:10.200469 16526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.126:40965 every 8 connection(s)
I20240503 14:05:10.204224 16527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:10.216032 16527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a: Bootstrap starting.
I20240503 14:05:10.220218 16527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:10.224433 16527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a: No bootstrap required, opened a new log
I20240503 14:05:10.226574 16527 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9862abea47943709ee653c229ff753a" member_type: VOTER }
I20240503 14:05:10.227063 16527 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:10.227280 16527 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9862abea47943709ee653c229ff753a, State: Initialized, Role: FOLLOWER
I20240503 14:05:10.227857 16527 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9862abea47943709ee653c229ff753a" member_type: VOTER }
I20240503 14:05:10.228315 16527 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:05:10.228533 16527 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:05:10.228778 16527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:10.233232 16527 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9862abea47943709ee653c229ff753a" member_type: VOTER }
I20240503 14:05:10.233800 16527 leader_election.cc:302] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9862abea47943709ee653c229ff753a; no voters:
I20240503 14:05:10.234956 16527 leader_election.cc:288] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:05:10.235420 16530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:10.236671 16530 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [term 1 LEADER]: Becoming Leader. State: Replica: d9862abea47943709ee653c229ff753a, State: Running, Role: LEADER
I20240503 14:05:10.237411 16530 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9862abea47943709ee653c229ff753a" member_type: VOTER }
I20240503 14:05:10.237756 16527 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:05:10.240415 16531 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9862abea47943709ee653c229ff753a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9862abea47943709ee653c229ff753a" member_type: VOTER } }
I20240503 14:05:10.240424 16532 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9862abea47943709ee653c229ff753a. Latest consensus state: current_term: 1 leader_uuid: "d9862abea47943709ee653c229ff753a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9862abea47943709ee653c229ff753a" member_type: VOTER } }
I20240503 14:05:10.241197 16531 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [sys.catalog]: This master's current role is: LEADER
I20240503 14:05:10.241278 16532 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [sys.catalog]: This master's current role is: LEADER
I20240503 14:05:10.244256 16535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:05:10.249050 16535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:05:10.253479 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:05:10.257723 16535 catalog_manager.cc:1347] Generated new cluster ID: 9e5e937f20b94623b549139abaf82923
I20240503 14:05:10.257959 16535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:05:10.284596 16535 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:05:10.285821 16535 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:05:10.301503 16535 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a: Generated new TSK 0
I20240503 14:05:10.302084 16535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:05:10.321180 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:10.328228 16548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:10.328826 16549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:10.330508 16551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:10.331557 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:10.332401 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:10.332587 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:10.332731 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745110332719 us; error 0 us; skew 500 ppm
I20240503 14:05:10.333262 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:10.335750 13945 webserver.cc:460] Webserver started at http://127.13.158.65:33491/ using document root <none> and password file <none>
I20240503 14:05:10.336205 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:10.336385 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:10.336627 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:10.337829 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-0-root/instance:
uuid: "9337e0d9fb154adc81ce90c8a48305bc"
format_stamp: "Formatted at 2024-05-03 14:05:10 on dist-test-slave-jm3r"
I20240503 14:05:10.343009 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20240503 14:05:10.346201 16556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.347092 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:05:10.347398 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-0-root
uuid: "9337e0d9fb154adc81ce90c8a48305bc"
format_stamp: "Formatted at 2024-05-03 14:05:10 on dist-test-slave-jm3r"
I20240503 14:05:10.347716 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:10.362095 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:10.363255 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:10.364687 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:10.367237 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:10.367439 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.367678 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:10.367836 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.407253 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.65:40429
I20240503 14:05:10.407353 16618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.65:40429 every 8 connection(s)
I20240503 14:05:10.411761 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:10.421303 16623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:10.421516 16624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:10.423260 16626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:10.424378 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:10.425052 16619 heartbeater.cc:347] Connected to a master server at 127.13.158.126:40965
I20240503 14:05:10.425295 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 14:05:10.425452 16619 heartbeater.cc:464] Registering TS with master...
W20240503 14:05:10.425557 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:10.425805 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745110425788 us; error 0 us; skew 500 ppm
I20240503 14:05:10.426514 16619 heartbeater.cc:511] Master 127.13.158.126:40965 requested a full tablet report, sending...
I20240503 14:05:10.426630 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:10.428575 16492 ts_manager.cc:194] Registered new tserver with Master: 9337e0d9fb154adc81ce90c8a48305bc (127.13.158.65:40429)
I20240503 14:05:10.429785 13945 webserver.cc:460] Webserver started at http://127.13.158.66:41363/ using document root <none> and password file <none>
I20240503 14:05:10.430243 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:10.430418 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:10.430331 16492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36652
I20240503 14:05:10.430840 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:10.432196 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-1-root/instance:
uuid: "f8c6df7847da463c909f06987f45ce69"
format_stamp: "Formatted at 2024-05-03 14:05:10 on dist-test-slave-jm3r"
I20240503 14:05:10.436636 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:05:10.439952 16631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.440644 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 14:05:10.440915 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-1-root
uuid: "f8c6df7847da463c909f06987f45ce69"
format_stamp: "Formatted at 2024-05-03 14:05:10 on dist-test-slave-jm3r"
I20240503 14:05:10.441197 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:10.466325 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:10.467519 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:10.468897 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:10.471096 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:10.471292 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.471501 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:10.471643 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.509234 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.66:36827
I20240503 14:05:10.509325 16693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.66:36827 every 8 connection(s)
I20240503 14:05:10.513779 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:10.521471 16697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:10.522369 16698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:10.524909 16694 heartbeater.cc:347] Connected to a master server at 127.13.158.126:40965
I20240503 14:05:10.525369 16694 heartbeater.cc:464] Registering TS with master...
W20240503 14:05:10.525326 16700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:10.526214 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:10.526400 16694 heartbeater.cc:511] Master 127.13.158.126:40965 requested a full tablet report, sending...
I20240503 14:05:10.527069 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:10.527319 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:10.527518 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745110527501 us; error 0 us; skew 500 ppm
I20240503 14:05:10.528084 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:10.528427 16492 ts_manager.cc:194] Registered new tserver with Master: f8c6df7847da463c909f06987f45ce69 (127.13.158.66:36827)
I20240503 14:05:10.529774 16492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36668
I20240503 14:05:10.530962 13945 webserver.cc:460] Webserver started at http://127.13.158.67:45297/ using document root <none> and password file <none>
I20240503 14:05:10.531453 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:10.531648 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:10.531909 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:10.533109 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-2-root/instance:
uuid: "13bff7d2cabe468b84ac6cd45255c095"
format_stamp: "Formatted at 2024-05-03 14:05:10 on dist-test-slave-jm3r"
I20240503 14:05:10.537654 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:05:10.541126 16705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.541904 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:05:10.542208 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-2-root
uuid: "13bff7d2cabe468b84ac6cd45255c095"
format_stamp: "Formatted at 2024-05-03 14:05:10 on dist-test-slave-jm3r"
I20240503 14:05:10.542539 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:10.564570 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:10.565675 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:10.567119 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:10.569483 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:10.569669 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.569914 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:10.570056 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.609103 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.67:44307
I20240503 14:05:10.609104 16767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.67:44307 every 8 connection(s)
I20240503 14:05:10.614099 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:10.623545 16771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:10.624706 16772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:10.625986 16768 heartbeater.cc:347] Connected to a master server at 127.13.158.126:40965
I20240503 14:05:10.626471 16768 heartbeater.cc:464] Registering TS with master...
I20240503 14:05:10.627188 16768 heartbeater.cc:511] Master 127.13.158.126:40965 requested a full tablet report, sending...
W20240503 14:05:10.627404 16774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:10.628767 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:10.629798 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 14:05:10.629860 16492 ts_manager.cc:194] Registered new tserver with Master: 13bff7d2cabe468b84ac6cd45255c095 (127.13.158.67:44307)
W20240503 14:05:10.630174 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:10.630446 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745110630401 us; error 0 us; skew 500 ppm
I20240503 14:05:10.631302 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:10.631863 16492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36670
I20240503 14:05:10.634245 13945 webserver.cc:460] Webserver started at http://127.13.158.68:34561/ using document root <none> and password file <none>
I20240503 14:05:10.634856 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:10.635057 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:10.635319 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:10.636497 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-3-root/instance:
uuid: "0fffc9aa23394a7e87cc477ce8afb1af"
format_stamp: "Formatted at 2024-05-03 14:05:10 on dist-test-slave-jm3r"
I20240503 14:05:10.641225 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:05:10.644703 16779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.645483 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:05:10.645771 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-3-root
uuid: "0fffc9aa23394a7e87cc477ce8afb1af"
format_stamp: "Formatted at 2024-05-03 14:05:10 on dist-test-slave-jm3r"
I20240503 14:05:10.646059 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714745036514908-13945-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:10.659080 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:10.660269 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:10.661681 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:10.663928 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:10.664130 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.664345 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:10.664494 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:10.703439 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.68:41739
I20240503 14:05:10.703550 16841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.68:41739 every 8 connection(s)
I20240503 14:05:10.716255 16842 heartbeater.cc:347] Connected to a master server at 127.13.158.126:40965
I20240503 14:05:10.716635 16842 heartbeater.cc:464] Registering TS with master...
I20240503 14:05:10.717355 16842 heartbeater.cc:511] Master 127.13.158.126:40965 requested a full tablet report, sending...
I20240503 14:05:10.719210 16492 ts_manager.cc:194] Registered new tserver with Master: 0fffc9aa23394a7e87cc477ce8afb1af (127.13.158.68:41739)
I20240503 14:05:10.720216 13945 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.013709547s
I20240503 14:05:10.720750 16492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36680
I20240503 14:05:11.433094 16619 heartbeater.cc:503] Master 127.13.158.126:40965 was elected leader, sending a full tablet report...
I20240503 14:05:11.532225 16694 heartbeater.cc:503] Master 127.13.158.126:40965 was elected leader, sending a full tablet report...
I20240503 14:05:11.634783 16768 heartbeater.cc:503] Master 127.13.158.126:40965 was elected leader, sending a full tablet report...
I20240503 14:05:11.723798 16842 heartbeater.cc:503] Master 127.13.158.126:40965 was elected leader, sending a full tablet report...
I20240503 14:05:11.764780 13945 test_util.cc:268] Using random seed: 893673237
I20240503 14:05:11.785945 16492 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36696:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:05:11.831846 16659 tablet_service.cc:1467] Processing CreateTablet for tablet 225ae129ab50485db3b38464cef38d52 (DEFAULT_TABLE table=test-workload [id=a1f1b8f33a2544e1a44dfaef94d0dfeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:05:11.833415 16659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 225ae129ab50485db3b38464cef38d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:11.838853 16733 tablet_service.cc:1467] Processing CreateTablet for tablet 225ae129ab50485db3b38464cef38d52 (DEFAULT_TABLE table=test-workload [id=a1f1b8f33a2544e1a44dfaef94d0dfeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:05:11.840185 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 225ae129ab50485db3b38464cef38d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:11.843752 16807 tablet_service.cc:1467] Processing CreateTablet for tablet 225ae129ab50485db3b38464cef38d52 (DEFAULT_TABLE table=test-workload [id=a1f1b8f33a2544e1a44dfaef94d0dfeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:05:11.845014 16807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 225ae129ab50485db3b38464cef38d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:11.859544 16863 tablet_bootstrap.cc:492] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69: Bootstrap starting.
I20240503 14:05:11.860911 16864 tablet_bootstrap.cc:492] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095: Bootstrap starting.
I20240503 14:05:11.862702 16865 tablet_bootstrap.cc:492] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af: Bootstrap starting.
I20240503 14:05:11.866688 16864 tablet_bootstrap.cc:654] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:11.868639 16863 tablet_bootstrap.cc:654] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:11.868667 16865 tablet_bootstrap.cc:654] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:11.871593 16864 tablet_bootstrap.cc:492] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095: No bootstrap required, opened a new log
I20240503 14:05:11.872107 16864 ts_tablet_manager.cc:1367] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.005s
I20240503 14:05:11.874995 16865 tablet_bootstrap.cc:492] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af: No bootstrap required, opened a new log
I20240503 14:05:11.874954 16864 raft_consensus.cc:348] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c6df7847da463c909f06987f45ce69" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 36827 } } peers { permanent_uuid: "13bff7d2cabe468b84ac6cd45255c095" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 44307 } } peers { permanent_uuid: "0fffc9aa23394a7e87cc477ce8afb1af" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 41739 } }
I20240503 14:05:11.875654 16865 ts_tablet_manager.cc:1367] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240503 14:05:11.875885 16864 raft_consensus.cc:374] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:11.876211 16864 raft_consensus.cc:729] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13bff7d2cabe468b84ac6cd45255c095, State: Initialized, Role: FOLLOWER
I20240503 14:05:11.876991 16864 consensus_queue.cc:260] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c6df7847da463c909f06987f45ce69" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 36827 } } peers { permanent_uuid: "13bff7d2cabe468b84ac6cd45255c095" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 44307 } } peers { permanent_uuid: "0fffc9aa23394a7e87cc477ce8afb1af" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 41739 } }
I20240503 14:05:11.876332 16863 tablet_bootstrap.cc:492] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69: No bootstrap required, opened a new log
I20240503 14:05:11.878139 16863 ts_tablet_manager.cc:1367] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.006s
I20240503 14:05:11.878455 16865 raft_consensus.cc:348] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c6df7847da463c909f06987f45ce69" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 36827 } } peers { permanent_uuid: "13bff7d2cabe468b84ac6cd45255c095" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 44307 } } peers { permanent_uuid: "0fffc9aa23394a7e87cc477ce8afb1af" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 41739 } }
I20240503 14:05:11.879355 16865 raft_consensus.cc:374] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:11.879652 16865 raft_consensus.cc:729] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fffc9aa23394a7e87cc477ce8afb1af, State: Initialized, Role: FOLLOWER
I20240503 14:05:11.880062 16864 ts_tablet_manager.cc:1398] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240503 14:05:11.880487 16865 consensus_queue.cc:260] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c6df7847da463c909f06987f45ce69" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 36827 } } peers { permanent_uuid: "13bff7d2cabe468b84ac6cd45255c095" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 44307 } } peers { permanent_uuid: "0fffc9aa23394a7e87cc477ce8afb1af" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 41739 } }
I20240503 14:05:11.880887 16863 raft_consensus.cc:348] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c6df7847da463c909f06987f45ce69" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 36827 } } peers { permanent_uuid: "13bff7d2cabe468b84ac6cd45255c095" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 44307 } } peers { permanent_uuid: "0fffc9aa23394a7e87cc477ce8afb1af" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 41739 } }
I20240503 14:05:11.882915 16863 raft_consensus.cc:374] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:11.883255 16863 raft_consensus.cc:729] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c6df7847da463c909f06987f45ce69, State: Initialized, Role: FOLLOWER
I20240503 14:05:11.883332 16865 ts_tablet_manager.cc:1398] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 14:05:11.884131 16863 consensus_queue.cc:260] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c6df7847da463c909f06987f45ce69" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 36827 } } peers { permanent_uuid: "13bff7d2cabe468b84ac6cd45255c095" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 44307 } } peers { permanent_uuid: "0fffc9aa23394a7e87cc477ce8afb1af" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 41739 } }
I20240503 14:05:11.887221 16863 ts_tablet_manager.cc:1398] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
W20240503 14:05:11.958571 16843 tablet.cc:2369] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:05:12.035797 16871 raft_consensus.cc:482] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:12.036427 16871 raft_consensus.cc:504] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c6df7847da463c909f06987f45ce69" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 36827 } } peers { permanent_uuid: "13bff7d2cabe468b84ac6cd45255c095" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 44307 } } peers { permanent_uuid: "0fffc9aa23394a7e87cc477ce8afb1af" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 41739 } }
I20240503 14:05:12.038372 16871 leader_election.cc:288] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13bff7d2cabe468b84ac6cd45255c095 (127.13.158.67:44307), 0fffc9aa23394a7e87cc477ce8afb1af (127.13.158.68:41739)
I20240503 14:05:12.049347 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225ae129ab50485db3b38464cef38d52" candidate_uuid: "f8c6df7847da463c909f06987f45ce69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13bff7d2cabe468b84ac6cd45255c095" is_pre_election: true
I20240503 14:05:12.050065 16743 raft_consensus.cc:2454] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c6df7847da463c909f06987f45ce69 in term 0.
I20240503 14:05:12.051178 16633 leader_election.cc:302] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13bff7d2cabe468b84ac6cd45255c095, f8c6df7847da463c909f06987f45ce69; no voters:
I20240503 14:05:12.051928 16871 raft_consensus.cc:2781] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:12.052206 16871 raft_consensus.cc:482] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:12.052251 16817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225ae129ab50485db3b38464cef38d52" candidate_uuid: "f8c6df7847da463c909f06987f45ce69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fffc9aa23394a7e87cc477ce8afb1af" is_pre_election: true
I20240503 14:05:12.052528 16871 raft_consensus.cc:3037] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:12.052906 16817 raft_consensus.cc:2454] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c6df7847da463c909f06987f45ce69 in term 0.
I20240503 14:05:12.057812 16871 raft_consensus.cc:504] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c6df7847da463c909f06987f45ce69" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 36827 } } peers { permanent_uuid: "13bff7d2cabe468b84ac6cd45255c095" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 44307 } } peers { permanent_uuid: "0fffc9aa23394a7e87cc477ce8afb1af" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 41739 } }
I20240503 14:05:12.059530 16871 leader_election.cc:288] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [CANDIDATE]: Term 1 election: Requested vote from peers 13bff7d2cabe468b84ac6cd45255c095 (127.13.158.67:44307), 0fffc9aa23394a7e87cc477ce8afb1af (127.13.158.68:41739)
I20240503 14:05:12.060329 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225ae129ab50485db3b38464cef38d52" candidate_uuid: "f8c6df7847da463c909f06987f45ce69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13bff7d2cabe468b84ac6cd45255c095"
I20240503 14:05:12.060544 16817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225ae129ab50485db3b38464cef38d52" candidate_uuid: "f8c6df7847da463c909f06987f45ce69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fffc9aa23394a7e87cc477ce8afb1af"
I20240503 14:05:12.060817 16743 raft_consensus.cc:3037] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:12.061041 16817 raft_consensus.cc:3037] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:12.065945 16743 raft_consensus.cc:2454] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c6df7847da463c909f06987f45ce69 in term 1.
I20240503 14:05:12.066282 16817 raft_consensus.cc:2454] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c6df7847da463c909f06987f45ce69 in term 1.
I20240503 14:05:12.067055 16633 leader_election.cc:302] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13bff7d2cabe468b84ac6cd45255c095, f8c6df7847da463c909f06987f45ce69; no voters:
I20240503 14:05:12.067727 16871 raft_consensus.cc:2781] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:12.068899 16871 raft_consensus.cc:686] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 1 LEADER]: Becoming Leader. State: Replica: f8c6df7847da463c909f06987f45ce69, State: Running, Role: LEADER
I20240503 14:05:12.069651 16871 consensus_queue.cc:237] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c6df7847da463c909f06987f45ce69" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 36827 } } peers { permanent_uuid: "13bff7d2cabe468b84ac6cd45255c095" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 44307 } } peers { permanent_uuid: "0fffc9aa23394a7e87cc477ce8afb1af" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 41739 } }
I20240503 14:05:12.077512 16490 catalog_manager.cc:5484] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c6df7847da463c909f06987f45ce69 (127.13.158.66). New cstate: current_term: 1 leader_uuid: "f8c6df7847da463c909f06987f45ce69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c6df7847da463c909f06987f45ce69" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 36827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13bff7d2cabe468b84ac6cd45255c095" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 44307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fffc9aa23394a7e87cc477ce8afb1af" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 41739 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:12.253556 16545 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:12.255558 16545 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:12.256001 16545 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:12.553236 16871 consensus_queue.cc:1035] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13bff7d2cabe468b84ac6cd45255c095" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 44307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:12.565886 16880 consensus_queue.cc:1035] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fffc9aa23394a7e87cc477ce8afb1af" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:14.257045 16545 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:14.258538 16545 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:14.258862 16545 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:16.259850 16545 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:16.261119 16545 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:16.261415 16545 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:18.262528 16545 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:18.263957 16545 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:18.264248 16545 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:20.265229 16545 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:20.266700 16545 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:20.267024 16545 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:22.267925 16545 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:22.269517 16545 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:22.269843 16545 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:23.153996 13945 tablet_server.cc:178] TabletServer@127.13.158.65:0 shutting down...
I20240503 14:05:23.180639 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:23.197513 13945 tablet_server.cc:194] TabletServer@127.13.158.65:0 shutdown complete.
I20240503 14:05:23.208717 13945 tablet_server.cc:178] TabletServer@127.13.158.66:0 shutting down...
I20240503 14:05:23.230101 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:23.230859 13945 tablet_replica.cc:330] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69: stopping tablet replica
I20240503 14:05:23.231490 13945 raft_consensus.cc:2229] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:05:23.232474 13945 raft_consensus.cc:2258] T 225ae129ab50485db3b38464cef38d52 P f8c6df7847da463c909f06987f45ce69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:23.253170 13945 tablet_server.cc:194] TabletServer@127.13.158.66:0 shutdown complete.
I20240503 14:05:23.263955 13945 tablet_server.cc:178] TabletServer@127.13.158.67:0 shutting down...
I20240503 14:05:23.283516 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:23.284201 13945 tablet_replica.cc:330] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095: stopping tablet replica
I20240503 14:05:23.284775 13945 raft_consensus.cc:2229] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:23.285248 13945 raft_consensus.cc:2258] T 225ae129ab50485db3b38464cef38d52 P 13bff7d2cabe468b84ac6cd45255c095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:23.304669 13945 tablet_server.cc:194] TabletServer@127.13.158.67:0 shutdown complete.
I20240503 14:05:23.314563 13945 tablet_server.cc:178] TabletServer@127.13.158.68:0 shutting down...
I20240503 14:05:23.333580 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:23.334224 13945 tablet_replica.cc:330] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af: stopping tablet replica
I20240503 14:05:23.334813 13945 raft_consensus.cc:2229] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:23.335243 13945 raft_consensus.cc:2258] T 225ae129ab50485db3b38464cef38d52 P 0fffc9aa23394a7e87cc477ce8afb1af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:23.354230 13945 tablet_server.cc:194] TabletServer@127.13.158.68:0 shutdown complete.
I20240503 14:05:23.368391 13945 master.cc:537] Master@127.13.158.126:40965 shutting down...
I20240503 14:05:23.385654 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:05:23.386273 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:23.386607 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P d9862abea47943709ee653c229ff753a: stopping tablet replica
I20240503 14:05:23.404780 13945 master.cc:552] Master@127.13.158.126:40965 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (13331 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240503 14:05:23.438596 13945 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.13.158.126:35927
I20240503 14:05:23.439728 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:23.446036 16886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:23.446966 16887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:23.447783 16889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:23.448463 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:23.449204 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:23.449396 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:23.449533 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745123449515 us; error 0 us; skew 500 ppm
I20240503 14:05:23.450008 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:23.458137 13945 webserver.cc:460] Webserver started at http://127.13.158.126:34489/ using document root <none> and password file <none>
I20240503 14:05:23.458832 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:23.459003 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:23.459226 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:23.460438 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/master-0-root/instance:
uuid: "06550499d5fb4d0ea745d6703f66dc3f"
format_stamp: "Formatted at 2024-05-03 14:05:23 on dist-test-slave-jm3r"
I20240503 14:05:23.465099 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:05:23.468443 16894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:23.469314 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 14:05:23.469606 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/master-0-root
uuid: "06550499d5fb4d0ea745d6703f66dc3f"
format_stamp: "Formatted at 2024-05-03 14:05:23 on dist-test-slave-jm3r"
I20240503 14:05:23.469884 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:23.506265 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:23.507551 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:23.548022 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.126:35927
I20240503 14:05:23.548138 16945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.126:35927 every 8 connection(s)
I20240503 14:05:23.552243 16946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:23.564318 16946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f: Bootstrap starting.
I20240503 14:05:23.568608 16946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:23.573038 16946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f: No bootstrap required, opened a new log
I20240503 14:05:23.575178 16946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06550499d5fb4d0ea745d6703f66dc3f" member_type: VOTER }
I20240503 14:05:23.575672 16946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:23.575903 16946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06550499d5fb4d0ea745d6703f66dc3f, State: Initialized, Role: FOLLOWER
I20240503 14:05:23.576481 16946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06550499d5fb4d0ea745d6703f66dc3f" member_type: VOTER }
I20240503 14:05:23.576967 16946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:05:23.577179 16946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:05:23.577433 16946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:23.581925 16946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06550499d5fb4d0ea745d6703f66dc3f" member_type: VOTER }
I20240503 14:05:23.582484 16946 leader_election.cc:302] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06550499d5fb4d0ea745d6703f66dc3f; no voters:
I20240503 14:05:23.583631 16946 leader_election.cc:288] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:05:23.584046 16949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:23.585310 16949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [term 1 LEADER]: Becoming Leader. State: Replica: 06550499d5fb4d0ea745d6703f66dc3f, State: Running, Role: LEADER
I20240503 14:05:23.586047 16949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06550499d5fb4d0ea745d6703f66dc3f" member_type: VOTER }
I20240503 14:05:23.586383 16946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:05:23.589721 16951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06550499d5fb4d0ea745d6703f66dc3f. Latest consensus state: current_term: 1 leader_uuid: "06550499d5fb4d0ea745d6703f66dc3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06550499d5fb4d0ea745d6703f66dc3f" member_type: VOTER } }
I20240503 14:05:23.590443 16951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [sys.catalog]: This master's current role is: LEADER
I20240503 14:05:23.589699 16950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06550499d5fb4d0ea745d6703f66dc3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06550499d5fb4d0ea745d6703f66dc3f" member_type: VOTER } }
I20240503 14:05:23.591142 16950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [sys.catalog]: This master's current role is: LEADER
I20240503 14:05:23.593004 16955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:05:23.597869 16955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:05:23.602782 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:05:23.607283 16955 catalog_manager.cc:1347] Generated new cluster ID: 096ad0f1a43f49bcbb7a259b555c7576
I20240503 14:05:23.607570 16955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:05:23.619765 16955 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:05:23.621115 16955 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:05:23.635022 16955 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f: Generated new TSK 0
I20240503 14:05:23.635675 16955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:05:23.669183 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:23.675805 16967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:23.676548 16968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:23.678369 16970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:23.679404 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:23.680166 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:23.680373 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:23.680500 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745123680486 us; error 0 us; skew 500 ppm
I20240503 14:05:23.681001 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:23.683388 13945 webserver.cc:460] Webserver started at http://127.13.158.65:35125/ using document root <none> and password file <none>
I20240503 14:05:23.683831 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:23.683995 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:23.684207 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:23.685312 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-0-root/instance:
uuid: "04e242bcd7664b1e8d07ec991bba7101"
format_stamp: "Formatted at 2024-05-03 14:05:23 on dist-test-slave-jm3r"
I20240503 14:05:23.690012 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240503 14:05:23.693440 16975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:23.694226 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:05:23.694566 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-0-root
uuid: "04e242bcd7664b1e8d07ec991bba7101"
format_stamp: "Formatted at 2024-05-03 14:05:23 on dist-test-slave-jm3r"
I20240503 14:05:23.694869 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:23.711728 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:23.712852 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:23.714488 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:23.716869 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:23.717077 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:23.717299 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:23.717473 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:23.756187 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.65:33987
I20240503 14:05:23.756280 17037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.65:33987 every 8 connection(s)
I20240503 14:05:23.760998 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:23.769747 17042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:23.770849 17043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:23.772167 17045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:23.773558 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:23.774363 17038 heartbeater.cc:347] Connected to a master server at 127.13.158.126:35927
I20240503 14:05:23.774498 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 14:05:23.774765 17038 heartbeater.cc:464] Registering TS with master...
W20240503 14:05:23.774780 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:23.775139 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745123775120 us; error 0 us; skew 500 ppm
I20240503 14:05:23.775534 17038 heartbeater.cc:511] Master 127.13.158.126:35927 requested a full tablet report, sending...
I20240503 14:05:23.775727 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:23.777619 16911 ts_manager.cc:194] Registered new tserver with Master: 04e242bcd7664b1e8d07ec991bba7101 (127.13.158.65:33987)
I20240503 14:05:23.778391 13945 webserver.cc:460] Webserver started at http://127.13.158.66:34171/ using document root <none> and password file <none>
I20240503 14:05:23.778909 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:23.779072 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:23.779327 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:23.779548 16911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38716
I20240503 14:05:23.780715 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-1-root/instance:
uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b"
format_stamp: "Formatted at 2024-05-03 14:05:23 on dist-test-slave-jm3r"
I20240503 14:05:23.785061 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:05:23.788326 17050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:23.789064 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:05:23.789340 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-1-root
uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b"
format_stamp: "Formatted at 2024-05-03 14:05:23 on dist-test-slave-jm3r"
I20240503 14:05:23.789597 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:23.799139 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:23.800269 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:23.801631 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:23.803867 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:23.804049 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:23.804279 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:23.804416 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:23.842841 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.66:38385
I20240503 14:05:23.842929 17112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.66:38385 every 8 connection(s)
I20240503 14:05:23.847796 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:23.856637 17116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:23.867617 17117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:23.869172 17113 heartbeater.cc:347] Connected to a master server at 127.13.158.126:35927
I20240503 14:05:23.869539 17113 heartbeater.cc:464] Registering TS with master...
I20240503 14:05:23.870311 17113 heartbeater.cc:511] Master 127.13.158.126:35927 requested a full tablet report, sending...
I20240503 14:05:23.870559 13945 server_base.cc:1034] running on GCE node
W20240503 14:05:23.871946 17119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:23.872947 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 14:05:23.873008 16911 ts_manager.cc:194] Registered new tserver with Master: b6a7e3d7488d40fbb6d2486cd6ee891b (127.13.158.66:38385)
W20240503 14:05:23.873319 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:23.873567 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745123873545 us; error 0 us; skew 500 ppm
I20240503 14:05:23.874384 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:23.874996 16911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38720
I20240503 14:05:23.877882 13945 webserver.cc:460] Webserver started at http://127.13.158.67:37457/ using document root <none> and password file <none>
I20240503 14:05:23.878461 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:23.878639 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:23.878875 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:23.880039 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-2-root/instance:
uuid: "b61abd0e70174c58b16cffe3dd3a6fc1"
format_stamp: "Formatted at 2024-05-03 14:05:23 on dist-test-slave-jm3r"
I20240503 14:05:23.884672 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:05:23.888111 17124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:23.888916 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:05:23.889187 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-2-root
uuid: "b61abd0e70174c58b16cffe3dd3a6fc1"
format_stamp: "Formatted at 2024-05-03 14:05:23 on dist-test-slave-jm3r"
I20240503 14:05:23.889465 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714745036514908-13945-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:23.899565 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:23.900683 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:23.902122 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:23.904363 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:23.904563 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:23.904800 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:23.904934 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:23.943284 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.67:38293
I20240503 14:05:23.943377 17186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.67:38293 every 8 connection(s)
I20240503 14:05:23.956121 17187 heartbeater.cc:347] Connected to a master server at 127.13.158.126:35927
I20240503 14:05:23.956494 17187 heartbeater.cc:464] Registering TS with master...
I20240503 14:05:23.957182 17187 heartbeater.cc:511] Master 127.13.158.126:35927 requested a full tablet report, sending...
I20240503 14:05:23.959007 16911 ts_manager.cc:194] Registered new tserver with Master: b61abd0e70174c58b16cffe3dd3a6fc1 (127.13.158.67:38293)
I20240503 14:05:23.959766 13945 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013463896s
I20240503 14:05:23.960384 16911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38732
I20240503 14:05:24.782007 17038 heartbeater.cc:503] Master 127.13.158.126:35927 was elected leader, sending a full tablet report...
I20240503 14:05:24.878118 17113 heartbeater.cc:503] Master 127.13.158.126:35927 was elected leader, sending a full tablet report...
I20240503 14:05:24.963079 17187 heartbeater.cc:503] Master 127.13.158.126:35927 was elected leader, sending a full tablet report...
I20240503 14:05:25.003648 13945 test_util.cc:268] Using random seed: 906912106
I20240503 14:05:25.025292 16911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38742:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:05:25.027261 16911 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:05:25.089476 17078 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4e5b2138d6465e98588ea804b971c7 (DEFAULT_TABLE table=test-workload [id=b7a329ad8aa047dfbb6f0f8a3d90bd7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:05:25.090348 17003 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4e5b2138d6465e98588ea804b971c7 (DEFAULT_TABLE table=test-workload [id=b7a329ad8aa047dfbb6f0f8a3d90bd7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:05:25.090853 17078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4e5b2138d6465e98588ea804b971c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:25.091549 17003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4e5b2138d6465e98588ea804b971c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:25.091760 17152 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4e5b2138d6465e98588ea804b971c7 (DEFAULT_TABLE table=test-workload [id=b7a329ad8aa047dfbb6f0f8a3d90bd7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:05:25.092941 17152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4e5b2138d6465e98588ea804b971c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:25.108896 17208 tablet_bootstrap.cc:492] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101: Bootstrap starting.
I20240503 14:05:25.111765 17209 tablet_bootstrap.cc:492] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b: Bootstrap starting.
I20240503 14:05:25.114352 17210 tablet_bootstrap.cc:492] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1: Bootstrap starting.
I20240503 14:05:25.114709 17208 tablet_bootstrap.cc:654] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:25.118018 17209 tablet_bootstrap.cc:654] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:25.119530 17210 tablet_bootstrap.cc:654] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:25.123307 17208 tablet_bootstrap.cc:492] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101: No bootstrap required, opened a new log
I20240503 14:05:25.123800 17208 ts_tablet_manager.cc:1367] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240503 14:05:25.124718 17209 tablet_bootstrap.cc:492] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b: No bootstrap required, opened a new log
I20240503 14:05:25.124873 17210 tablet_bootstrap.cc:492] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1: No bootstrap required, opened a new log
I20240503 14:05:25.125136 17209 ts_tablet_manager.cc:1367] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240503 14:05:25.125375 17210 ts_tablet_manager.cc:1367] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240503 14:05:25.126529 17208 raft_consensus.cc:348] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38385 } } peers { permanent_uuid: "04e242bcd7664b1e8d07ec991bba7101" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 33987 } } peers { permanent_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 38293 } }
I20240503 14:05:25.127362 17208 raft_consensus.cc:374] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:25.127214 17209 raft_consensus.cc:348] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38385 } } peers { permanent_uuid: "04e242bcd7664b1e8d07ec991bba7101" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 33987 } } peers { permanent_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 38293 } }
I20240503 14:05:25.127724 17208 raft_consensus.cc:729] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e242bcd7664b1e8d07ec991bba7101, State: Initialized, Role: FOLLOWER
I20240503 14:05:25.127918 17209 raft_consensus.cc:374] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:25.128235 17209 raft_consensus.cc:729] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a7e3d7488d40fbb6d2486cd6ee891b, State: Initialized, Role: FOLLOWER
I20240503 14:05:25.128059 17210 raft_consensus.cc:348] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38385 } } peers { permanent_uuid: "04e242bcd7664b1e8d07ec991bba7101" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 33987 } } peers { permanent_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 38293 } }
I20240503 14:05:25.128520 17208 consensus_queue.cc:260] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38385 } } peers { permanent_uuid: "04e242bcd7664b1e8d07ec991bba7101" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 33987 } } peers { permanent_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 38293 } }
I20240503 14:05:25.128921 17210 raft_consensus.cc:374] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:25.128943 17209 consensus_queue.cc:260] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38385 } } peers { permanent_uuid: "04e242bcd7664b1e8d07ec991bba7101" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 33987 } } peers { permanent_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 38293 } }
I20240503 14:05:25.129492 17210 raft_consensus.cc:729] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b61abd0e70174c58b16cffe3dd3a6fc1, State: Initialized, Role: FOLLOWER
I20240503 14:05:25.130308 17210 consensus_queue.cc:260] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38385 } } peers { permanent_uuid: "04e242bcd7664b1e8d07ec991bba7101" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 33987 } } peers { permanent_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 38293 } }
I20240503 14:05:25.139299 17209 ts_tablet_manager.cc:1398] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b: Time spent starting tablet: real 0.014s user 0.006s sys 0.009s
I20240503 14:05:25.140762 17210 ts_tablet_manager.cc:1398] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20240503 14:05:25.139426 17208 ts_tablet_manager.cc:1398] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20240503 14:05:25.254565 17214 raft_consensus.cc:482] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:25.255057 17214 raft_consensus.cc:504] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38385 } } peers { permanent_uuid: "04e242bcd7664b1e8d07ec991bba7101" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 33987 } } peers { permanent_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 38293 } }
I20240503 14:05:25.257283 17214 leader_election.cc:288] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6a7e3d7488d40fbb6d2486cd6ee891b (127.13.158.66:38385), b61abd0e70174c58b16cffe3dd3a6fc1 (127.13.158.67:38293)
I20240503 14:05:25.269903 17162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4e5b2138d6465e98588ea804b971c7" candidate_uuid: "04e242bcd7664b1e8d07ec991bba7101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1" is_pre_election: true
I20240503 14:05:25.269858 17088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4e5b2138d6465e98588ea804b971c7" candidate_uuid: "04e242bcd7664b1e8d07ec991bba7101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b" is_pre_election: true
I20240503 14:05:25.270917 17162 raft_consensus.cc:2454] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04e242bcd7664b1e8d07ec991bba7101 in term 0.
I20240503 14:05:25.271008 17088 raft_consensus.cc:2454] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04e242bcd7664b1e8d07ec991bba7101 in term 0.
I20240503 14:05:25.272218 16978 leader_election.cc:302] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e242bcd7664b1e8d07ec991bba7101, b61abd0e70174c58b16cffe3dd3a6fc1; no voters:
I20240503 14:05:25.272949 17214 raft_consensus.cc:2781] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:25.273303 17214 raft_consensus.cc:482] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:25.273582 17214 raft_consensus.cc:3037] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:25.278822 17214 raft_consensus.cc:504] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38385 } } peers { permanent_uuid: "04e242bcd7664b1e8d07ec991bba7101" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 33987 } } peers { permanent_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 38293 } }
I20240503 14:05:25.280439 17214 leader_election.cc:288] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [CANDIDATE]: Term 1 election: Requested vote from peers b6a7e3d7488d40fbb6d2486cd6ee891b (127.13.158.66:38385), b61abd0e70174c58b16cffe3dd3a6fc1 (127.13.158.67:38293)
I20240503 14:05:25.281682 17162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4e5b2138d6465e98588ea804b971c7" candidate_uuid: "04e242bcd7664b1e8d07ec991bba7101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1"
I20240503 14:05:25.281695 17088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4e5b2138d6465e98588ea804b971c7" candidate_uuid: "04e242bcd7664b1e8d07ec991bba7101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b"
I20240503 14:05:25.282263 17162 raft_consensus.cc:3037] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:25.282397 17088 raft_consensus.cc:3037] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:25.287701 17088 raft_consensus.cc:2454] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04e242bcd7664b1e8d07ec991bba7101 in term 1.
I20240503 14:05:25.287690 17162 raft_consensus.cc:2454] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04e242bcd7664b1e8d07ec991bba7101 in term 1.
I20240503 14:05:25.288971 16976 leader_election.cc:302] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e242bcd7664b1e8d07ec991bba7101, b6a7e3d7488d40fbb6d2486cd6ee891b; no voters:
I20240503 14:05:25.289680 17214 raft_consensus.cc:2781] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:25.290982 17214 raft_consensus.cc:686] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 1 LEADER]: Becoming Leader. State: Replica: 04e242bcd7664b1e8d07ec991bba7101, State: Running, Role: LEADER
I20240503 14:05:25.292445 17214 consensus_queue.cc:237] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38385 } } peers { permanent_uuid: "04e242bcd7664b1e8d07ec991bba7101" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 33987 } } peers { permanent_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 38293 } }
I20240503 14:05:25.299110 16911 catalog_manager.cc:5484] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04e242bcd7664b1e8d07ec991bba7101 (127.13.158.65). New cstate: current_term: 1 leader_uuid: "04e242bcd7664b1e8d07ec991bba7101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04e242bcd7664b1e8d07ec991bba7101" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 33987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:25.600734 16959 placement_policy_util.cc:407] tablet 2f4e5b2138d6465e98588ea804b971c7: detected majority of replicas (2 of 3) at location L0
I20240503 14:05:25.602808 16964 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:25.604737 16964 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:25.605269 16964 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:25.890574 17221 consensus_queue.cc:1035] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61abd0e70174c58b16cffe3dd3a6fc1" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:25.894392 17222 consensus_queue.cc:1035] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a7e3d7488d40fbb6d2486cd6ee891b" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:26.610293 16959 placement_policy_util.cc:407] tablet 2f4e5b2138d6465e98588ea804b971c7: detected majority of replicas (2 of 3) at location L0
I20240503 14:05:27.606806 16964 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:27.608562 16964 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:27.608877 16964 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:27.618317 16959 placement_policy_util.cc:407] tablet 2f4e5b2138d6465e98588ea804b971c7: detected majority of replicas (2 of 3) at location L0
I20240503 14:05:28.626169 16959 placement_policy_util.cc:407] tablet 2f4e5b2138d6465e98588ea804b971c7: detected majority of replicas (2 of 3) at location L0
I20240503 14:05:29.609737 16964 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:29.611022 16964 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:29.611295 16964 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:29.633874 16959 placement_policy_util.cc:407] tablet 2f4e5b2138d6465e98588ea804b971c7: detected majority of replicas (2 of 3) at location L0
I20240503 14:05:30.641580 16959 placement_policy_util.cc:407] tablet 2f4e5b2138d6465e98588ea804b971c7: detected majority of replicas (2 of 3) at location L0
I20240503 14:05:31.612334 16964 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:31.615326 16964 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:31.615989 16964 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:31.649374 16959 placement_policy_util.cc:407] tablet 2f4e5b2138d6465e98588ea804b971c7: detected majority of replicas (2 of 3) at location L0
I20240503 14:05:32.657903 16959 placement_policy_util.cc:407] tablet 2f4e5b2138d6465e98588ea804b971c7: detected majority of replicas (2 of 3) at location L0
W20240503 14:05:32.800923 17183 debug-util.cc:398] Leaking SignalData structure 0x7b08001aff20 after lost signal to thread 13948
W20240503 14:05:32.802357 17183 debug-util.cc:398] Leaking SignalData structure 0x7b080010c580 after lost signal to thread 16945
W20240503 14:05:32.840667 17183 debug-util.cc:398] Leaking SignalData structure 0x7b08000f3840 after lost signal to thread 17037
W20240503 14:05:32.843147 17183 debug-util.cc:398] Leaking SignalData structure 0x7b0800241720 after lost signal to thread 17112
W20240503 14:05:32.844439 17183 debug-util.cc:398] Leaking SignalData structure 0x7b080012a800 after lost signal to thread 17186
W20240503 14:05:33.388716 17034 debug-util.cc:398] Leaking SignalData structure 0x7b08001e82c0 after lost signal to thread 13948
W20240503 14:05:33.389889 17034 debug-util.cc:398] Leaking SignalData structure 0x7b08001e45e0 after lost signal to thread 16945
W20240503 14:05:33.391322 17034 debug-util.cc:398] Leaking SignalData structure 0x7b08000e0e80 after lost signal to thread 17037
W20240503 14:05:33.392510 17034 debug-util.cc:398] Leaking SignalData structure 0x7b0800109520 after lost signal to thread 17112
W20240503 14:05:33.393677 17034 debug-util.cc:398] Leaking SignalData structure 0x7b08000642e0 after lost signal to thread 17186
I20240503 14:05:33.617151 16964 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:33.619064 16964 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:33.619496 16964 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:33.774834 16959 placement_policy_util.cc:407] tablet 2f4e5b2138d6465e98588ea804b971c7: detected majority of replicas (2 of 3) at location L0
I20240503 14:05:34.784850 16959 placement_policy_util.cc:407] tablet 2f4e5b2138d6465e98588ea804b971c7: detected majority of replicas (2 of 3) at location L0
I20240503 14:05:35.620535 16964 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:35.622244 16964 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:35.622675 16964 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:35.793408 16959 placement_policy_util.cc:407] tablet 2f4e5b2138d6465e98588ea804b971c7: detected majority of replicas (2 of 3) at location L0
I20240503 14:05:36.573120 13945 tablet_server.cc:178] TabletServer@127.13.158.65:0 shutting down...
I20240503 14:05:36.598574 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:36.599301 13945 tablet_replica.cc:330] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101: stopping tablet replica
I20240503 14:05:36.599854 13945 raft_consensus.cc:2229] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:05:36.600777 13945 raft_consensus.cc:2258] T 2f4e5b2138d6465e98588ea804b971c7 P 04e242bcd7664b1e8d07ec991bba7101 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:36.638267 13945 tablet_server.cc:194] TabletServer@127.13.158.65:0 shutdown complete.
I20240503 14:05:36.650214 13945 tablet_server.cc:178] TabletServer@127.13.158.66:0 shutting down...
I20240503 14:05:36.670928 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:36.671602 13945 tablet_replica.cc:330] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b: stopping tablet replica
I20240503 14:05:36.672150 13945 raft_consensus.cc:2229] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:36.672626 13945 raft_consensus.cc:2258] T 2f4e5b2138d6465e98588ea804b971c7 P b6a7e3d7488d40fbb6d2486cd6ee891b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:36.691985 13945 tablet_server.cc:194] TabletServer@127.13.158.66:0 shutdown complete.
I20240503 14:05:36.702227 13945 tablet_server.cc:178] TabletServer@127.13.158.67:0 shutting down...
I20240503 14:05:36.721443 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:36.722128 13945 tablet_replica.cc:330] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1: stopping tablet replica
I20240503 14:05:36.722743 13945 raft_consensus.cc:2229] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:36.723234 13945 raft_consensus.cc:2258] T 2f4e5b2138d6465e98588ea804b971c7 P b61abd0e70174c58b16cffe3dd3a6fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:36.757937 13945 tablet_server.cc:194] TabletServer@127.13.158.67:0 shutdown complete.
I20240503 14:05:36.768379 13945 master.cc:537] Master@127.13.158.126:35927 shutting down...
I20240503 14:05:36.786733 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:05:36.787380 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:36.787714 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 06550499d5fb4d0ea745d6703f66dc3f: stopping tablet replica
I20240503 14:05:36.806839 13945 master.cc:552] Master@127.13.158.126:35927 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (13401 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:516: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] AutoRebalancerTest.TestMaxMovesPerServer (7 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240503 14:05:36.847554 13945 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.13.158.126:46229
I20240503 14:05:36.848632 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:36.855439 17231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:36.856096 17232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:36.857170 17234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:36.857997 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:36.858907 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:36.859110 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:36.859277 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745136859260 us; error 0 us; skew 500 ppm
I20240503 14:05:36.859758 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:36.867856 13945 webserver.cc:460] Webserver started at http://127.13.158.126:43631/ using document root <none> and password file <none>
I20240503 14:05:36.868404 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:36.868593 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:36.868881 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:36.870182 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/master-0-root/instance:
uuid: "304b2a18064d4920804af80fc1555430"
format_stamp: "Formatted at 2024-05-03 14:05:36 on dist-test-slave-jm3r"
I20240503 14:05:36.874876 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:05:36.878381 17239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:36.879235 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:05:36.879536 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/master-0-root
uuid: "304b2a18064d4920804af80fc1555430"
format_stamp: "Formatted at 2024-05-03 14:05:36 on dist-test-slave-jm3r"
I20240503 14:05:36.879920 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:36.903275 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:36.904515 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:36.943717 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.126:46229
I20240503 14:05:36.943812 17290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.126:46229 every 8 connection(s)
I20240503 14:05:36.947731 17291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:36.958876 17291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430: Bootstrap starting.
I20240503 14:05:36.964474 17291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:36.970703 17291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430: No bootstrap required, opened a new log
I20240503 14:05:36.972793 17291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304b2a18064d4920804af80fc1555430" member_type: VOTER }
I20240503 14:05:36.973294 17291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:36.973472 17291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304b2a18064d4920804af80fc1555430, State: Initialized, Role: FOLLOWER
I20240503 14:05:36.974020 17291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304b2a18064d4920804af80fc1555430" member_type: VOTER }
I20240503 14:05:36.974527 17291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:05:36.974746 17291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:05:36.975055 17291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:36.981050 17291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304b2a18064d4920804af80fc1555430" member_type: VOTER }
I20240503 14:05:36.981693 17291 leader_election.cc:302] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 304b2a18064d4920804af80fc1555430; no voters:
I20240503 14:05:36.983462 17291 leader_election.cc:288] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:05:36.983737 17294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:36.985241 17294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [term 1 LEADER]: Becoming Leader. State: Replica: 304b2a18064d4920804af80fc1555430, State: Running, Role: LEADER
I20240503 14:05:36.986029 17294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304b2a18064d4920804af80fc1555430" member_type: VOTER }
I20240503 14:05:36.986577 17291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:05:36.991350 17296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 304b2a18064d4920804af80fc1555430. Latest consensus state: current_term: 1 leader_uuid: "304b2a18064d4920804af80fc1555430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304b2a18064d4920804af80fc1555430" member_type: VOTER } }
I20240503 14:05:36.992065 17296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [sys.catalog]: This master's current role is: LEADER
I20240503 14:05:36.991533 17295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "304b2a18064d4920804af80fc1555430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304b2a18064d4920804af80fc1555430" member_type: VOTER } }
I20240503 14:05:36.993280 17295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [sys.catalog]: This master's current role is: LEADER
I20240503 14:05:36.994182 17302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:05:37.001673 17302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:05:37.002717 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:05:37.010561 17302 catalog_manager.cc:1347] Generated new cluster ID: 5f848e16a2fd42cc852656649188c9fe
I20240503 14:05:37.010905 17302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:05:37.056830 17302 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:05:37.058817 17302 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:05:37.075922 17302 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430: Generated new TSK 0
I20240503 14:05:37.076673 17302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:05:37.089196 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:37.097925 17313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:37.098492 17314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:37.099406 17316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:37.100360 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:37.101406 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:37.101615 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:37.101750 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745137101740 us; error 0 us; skew 500 ppm
I20240503 14:05:37.102275 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:37.105291 13945 webserver.cc:460] Webserver started at http://127.13.158.65:33999/ using document root <none> and password file <none>
I20240503 14:05:37.105818 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:37.106026 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:37.106307 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:37.108004 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-0-root/instance:
uuid: "30ac3c2f11a44d90ba10d615508b5e67"
format_stamp: "Formatted at 2024-05-03 14:05:37 on dist-test-slave-jm3r"
I20240503 14:05:37.113914 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 14:05:37.117877 17321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:37.118885 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:05:37.119216 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-0-root
uuid: "30ac3c2f11a44d90ba10d615508b5e67"
format_stamp: "Formatted at 2024-05-03 14:05:37 on dist-test-slave-jm3r"
I20240503 14:05:37.119503 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:37.145181 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:37.146476 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:37.148046 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:37.150794 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:37.151032 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:37.151280 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:37.151443 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:37.194851 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.65:36197
I20240503 14:05:37.194942 17383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.65:36197 every 8 connection(s)
I20240503 14:05:37.200337 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:37.208688 17388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:37.209368 17389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:37.215035 17391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:37.215698 17384 heartbeater.cc:347] Connected to a master server at 127.13.158.126:46229
I20240503 14:05:37.216341 17384 heartbeater.cc:464] Registering TS with master...
I20240503 14:05:37.216370 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:37.217396 17384 heartbeater.cc:511] Master 127.13.158.126:46229 requested a full tablet report, sending...
I20240503 14:05:37.217526 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:37.217795 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:37.217948 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745137217931 us; error 0 us; skew 500 ppm
I20240503 14:05:37.218762 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:37.220049 17256 ts_manager.cc:194] Registered new tserver with Master: 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65:36197)
I20240503 14:05:37.222541 17256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58810
I20240503 14:05:37.225440 13945 webserver.cc:460] Webserver started at http://127.13.158.66:46659/ using document root <none> and password file <none>
I20240503 14:05:37.226004 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:37.226209 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:37.226512 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:37.227924 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-1-root/instance:
uuid: "2171b09913424ab3b53d5397987eebcf"
format_stamp: "Formatted at 2024-05-03 14:05:37 on dist-test-slave-jm3r"
I20240503 14:05:37.232952 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:05:37.237377 17396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:37.238500 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:05:37.238941 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-1-root
uuid: "2171b09913424ab3b53d5397987eebcf"
format_stamp: "Formatted at 2024-05-03 14:05:37 on dist-test-slave-jm3r"
I20240503 14:05:37.239313 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:37.257443 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:37.258802 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:37.260282 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:37.262647 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:37.262862 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:37.263100 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:37.263249 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:37.307842 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.66:35177
I20240503 14:05:37.307958 17458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.66:35177 every 8 connection(s)
I20240503 14:05:37.312922 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:37.321484 17462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:37.322319 17463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:37.325311 17465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:37.326239 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:37.326809 17459 heartbeater.cc:347] Connected to a master server at 127.13.158.126:46229
I20240503 14:05:37.327168 17459 heartbeater.cc:464] Registering TS with master...
I20240503 14:05:37.327214 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:37.327522 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:37.327683 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745137327668 us; error 0 us; skew 500 ppm
I20240503 14:05:37.328033 17459 heartbeater.cc:511] Master 127.13.158.126:46229 requested a full tablet report, sending...
I20240503 14:05:37.328295 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:37.330278 17256 ts_manager.cc:194] Registered new tserver with Master: 2171b09913424ab3b53d5397987eebcf (127.13.158.66:35177)
I20240503 14:05:37.331480 13945 webserver.cc:460] Webserver started at http://127.13.158.67:34269/ using document root <none> and password file <none>
I20240503 14:05:37.332161 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:37.332405 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:37.332401 17256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58812
I20240503 14:05:37.332964 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:37.334355 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-2-root/instance:
uuid: "ab5460d695ad4c1881d5f82b4df8adbf"
format_stamp: "Formatted at 2024-05-03 14:05:37 on dist-test-slave-jm3r"
I20240503 14:05:37.339169 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:05:37.342916 17470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:37.343685 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:05:37.344002 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-2-root
uuid: "ab5460d695ad4c1881d5f82b4df8adbf"
format_stamp: "Formatted at 2024-05-03 14:05:37 on dist-test-slave-jm3r"
I20240503 14:05:37.344288 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:37.364928 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:37.366108 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:37.367815 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:37.370307 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:37.370568 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240503 14:05:37.370813 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:37.370970 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:37.411612 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.67:34965
I20240503 14:05:37.411715 17532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.67:34965 every 8 connection(s)
I20240503 14:05:37.425262 17533 heartbeater.cc:347] Connected to a master server at 127.13.158.126:46229
I20240503 14:05:37.425652 17533 heartbeater.cc:464] Registering TS with master...
I20240503 14:05:37.426415 17533 heartbeater.cc:511] Master 127.13.158.126:46229 requested a full tablet report, sending...
I20240503 14:05:37.428310 17256 ts_manager.cc:194] Registered new tserver with Master: ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67:34965)
I20240503 14:05:37.428489 13945 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013797454s
I20240503 14:05:37.430160 17256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58826
I20240503 14:05:38.225641 17384 heartbeater.cc:503] Master 127.13.158.126:46229 was elected leader, sending a full tablet report...
I20240503 14:05:38.335345 17459 heartbeater.cc:503] Master 127.13.158.126:46229 was elected leader, sending a full tablet report...
I20240503 14:05:38.433141 17533 heartbeater.cc:503] Master 127.13.158.126:46229 was elected leader, sending a full tablet report...
I20240503 14:05:38.461973 13945 test_util.cc:268] Using random seed: 920370430
I20240503 14:05:38.486124 17256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58836:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:05:38.488546 17256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:05:38.552798 17349 tablet_service.cc:1467] Processing CreateTablet for tablet 2032d05f926d4989ba2d00c1ea704f8a (DEFAULT_TABLE table=test-workload [id=13b3d0790bb342d5bbe9211b1f9acf0d]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 14:05:38.552888 17347 tablet_service.cc:1467] Processing CreateTablet for tablet da79a01e920f448384284e3d77da4293 (DEFAULT_TABLE table=test-workload [id=13b3d0790bb342d5bbe9211b1f9acf0d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 14:05:38.552703 17348 tablet_service.cc:1467] Processing CreateTablet for tablet a3c1915f91a94b4cb25df5f593a14797 (DEFAULT_TABLE table=test-workload [id=13b3d0790bb342d5bbe9211b1f9acf0d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 14:05:38.555431 17349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2032d05f926d4989ba2d00c1ea704f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:38.556010 17424 tablet_service.cc:1467] Processing CreateTablet for tablet 2032d05f926d4989ba2d00c1ea704f8a (DEFAULT_TABLE table=test-workload [id=13b3d0790bb342d5bbe9211b1f9acf0d]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 14:05:38.556502 17423 tablet_service.cc:1467] Processing CreateTablet for tablet a3c1915f91a94b4cb25df5f593a14797 (DEFAULT_TABLE table=test-workload [id=13b3d0790bb342d5bbe9211b1f9acf0d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 14:05:38.557085 17347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da79a01e920f448384284e3d77da4293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:38.557514 17424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2032d05f926d4989ba2d00c1ea704f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:38.558670 17348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c1915f91a94b4cb25df5f593a14797. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:38.559302 17423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c1915f91a94b4cb25df5f593a14797. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:38.558813 17422 tablet_service.cc:1467] Processing CreateTablet for tablet da79a01e920f448384284e3d77da4293 (DEFAULT_TABLE table=test-workload [id=13b3d0790bb342d5bbe9211b1f9acf0d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 14:05:38.561668 17422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da79a01e920f448384284e3d77da4293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:38.569792 17498 tablet_service.cc:1467] Processing CreateTablet for tablet 2032d05f926d4989ba2d00c1ea704f8a (DEFAULT_TABLE table=test-workload [id=13b3d0790bb342d5bbe9211b1f9acf0d]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 14:05:38.570576 17496 tablet_service.cc:1467] Processing CreateTablet for tablet da79a01e920f448384284e3d77da4293 (DEFAULT_TABLE table=test-workload [id=13b3d0790bb342d5bbe9211b1f9acf0d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 14:05:38.571321 17498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2032d05f926d4989ba2d00c1ea704f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:38.572814 17497 tablet_service.cc:1467] Processing CreateTablet for tablet a3c1915f91a94b4cb25df5f593a14797 (DEFAULT_TABLE table=test-workload [id=13b3d0790bb342d5bbe9211b1f9acf0d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 14:05:38.574194 17497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c1915f91a94b4cb25df5f593a14797. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:38.578838 17496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da79a01e920f448384284e3d77da4293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:38.601397 17553 tablet_bootstrap.cc:492] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf: Bootstrap starting.
I20240503 14:05:38.603170 17554 tablet_bootstrap.cc:492] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf: Bootstrap starting.
I20240503 14:05:38.608203 17553 tablet_bootstrap.cc:654] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:38.608942 17554 tablet_bootstrap.cc:654] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:38.617928 17557 tablet_bootstrap.cc:492] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67: Bootstrap starting.
I20240503 14:05:38.625349 17554 tablet_bootstrap.cc:492] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf: No bootstrap required, opened a new log
I20240503 14:05:38.626024 17554 ts_tablet_manager.cc:1367] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.009s
I20240503 14:05:38.629791 17557 tablet_bootstrap.cc:654] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:38.629027 17554 raft_consensus.cc:348] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:38.630728 17554 raft_consensus.cc:374] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:38.632009 17554 raft_consensus.cc:729] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5460d695ad4c1881d5f82b4df8adbf, State: Initialized, Role: FOLLOWER
I20240503 14:05:38.631685 17553 tablet_bootstrap.cc:492] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf: No bootstrap required, opened a new log
I20240503 14:05:38.632856 17554 consensus_queue.cc:260] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:38.633422 17553 ts_tablet_manager.cc:1367] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.004s
I20240503 14:05:38.636430 17553 raft_consensus.cc:348] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:38.637351 17553 raft_consensus.cc:374] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:38.637712 17553 raft_consensus.cc:729] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2171b09913424ab3b53d5397987eebcf, State: Initialized, Role: FOLLOWER
I20240503 14:05:38.637945 17554 ts_tablet_manager.cc:1398] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20240503 14:05:38.638594 17557 tablet_bootstrap.cc:492] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67: No bootstrap required, opened a new log
I20240503 14:05:38.639019 17554 tablet_bootstrap.cc:492] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf: Bootstrap starting.
I20240503 14:05:38.639209 17557 ts_tablet_manager.cc:1367] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.008s
I20240503 14:05:38.638674 17553 consensus_queue.cc:260] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:38.642341 17557 raft_consensus.cc:348] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:38.643306 17557 raft_consensus.cc:374] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:38.643656 17557 raft_consensus.cc:729] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ac3c2f11a44d90ba10d615508b5e67, State: Initialized, Role: FOLLOWER
I20240503 14:05:38.644636 17557 consensus_queue.cc:260] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:38.646883 17554 tablet_bootstrap.cc:654] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:38.657243 17557 ts_tablet_manager.cc:1398] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67: Time spent starting tablet: real 0.018s user 0.008s sys 0.008s
I20240503 14:05:38.657676 17553 ts_tablet_manager.cc:1398] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf: Time spent starting tablet: real 0.024s user 0.017s sys 0.004s
I20240503 14:05:38.658766 17554 tablet_bootstrap.cc:492] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf: No bootstrap required, opened a new log
I20240503 14:05:38.659488 17554 ts_tablet_manager.cc:1367] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.005s
I20240503 14:05:38.659770 17557 tablet_bootstrap.cc:492] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67: Bootstrap starting.
I20240503 14:05:38.658843 17553 tablet_bootstrap.cc:492] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf: Bootstrap starting.
I20240503 14:05:38.662313 17554 raft_consensus.cc:348] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.663300 17554 raft_consensus.cc:374] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:38.663666 17554 raft_consensus.cc:729] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5460d695ad4c1881d5f82b4df8adbf, State: Initialized, Role: FOLLOWER
I20240503 14:05:38.664410 17554 consensus_queue.cc:260] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.666589 17554 ts_tablet_manager.cc:1398] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240503 14:05:38.666623 17553 tablet_bootstrap.cc:654] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:38.668074 17560 raft_consensus.cc:482] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:38.668725 17560 raft_consensus.cc:504] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:38.673409 17554 tablet_bootstrap.cc:492] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf: Bootstrap starting.
I20240503 14:05:38.674690 17560 leader_election.cc:288] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67:34965), 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65:36197)
I20240503 14:05:38.676625 17557 tablet_bootstrap.cc:654] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:38.679781 17554 tablet_bootstrap.cc:654] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:38.682406 17553 tablet_bootstrap.cc:492] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf: No bootstrap required, opened a new log
I20240503 14:05:38.682989 17553 ts_tablet_manager.cc:1367] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.004s
I20240503 14:05:38.688169 17553 raft_consensus.cc:348] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.689007 17553 raft_consensus.cc:374] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:38.689136 17554 tablet_bootstrap.cc:492] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf: No bootstrap required, opened a new log
I20240503 14:05:38.690065 17554 ts_tablet_manager.cc:1367] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20240503 14:05:38.690944 17561 raft_consensus.cc:482] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:38.691424 17561 raft_consensus.cc:504] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:38.693063 17554 raft_consensus.cc:348] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.693907 17554 raft_consensus.cc:374] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:38.689353 17553 raft_consensus.cc:729] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2171b09913424ab3b53d5397987eebcf, State: Initialized, Role: FOLLOWER
I20240503 14:05:38.694195 17554 raft_consensus.cc:729] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5460d695ad4c1881d5f82b4df8adbf, State: Initialized, Role: FOLLOWER
I20240503 14:05:38.695009 17553 consensus_queue.cc:260] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.695780 17554 consensus_queue.cc:260] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.696945 17561 leader_election.cc:288] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2171b09913424ab3b53d5397987eebcf (127.13.158.66:35177), ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67:34965)
I20240503 14:05:38.698907 17553 ts_tablet_manager.cc:1398] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20240503 14:05:38.699863 17553 tablet_bootstrap.cc:492] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf: Bootstrap starting.
I20240503 14:05:38.705603 17553 tablet_bootstrap.cc:654] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:38.708165 17557 tablet_bootstrap.cc:492] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67: No bootstrap required, opened a new log
I20240503 14:05:38.705528 17554 ts_tablet_manager.cc:1398] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240503 14:05:38.708817 17557 ts_tablet_manager.cc:1367] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.015s
I20240503 14:05:38.714638 17557 raft_consensus.cc:348] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.715649 17557 raft_consensus.cc:374] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:38.715994 17557 raft_consensus.cc:729] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ac3c2f11a44d90ba10d615508b5e67, State: Initialized, Role: FOLLOWER
I20240503 14:05:38.716969 17557 consensus_queue.cc:260] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.730058 17557 ts_tablet_manager.cc:1398] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20240503 14:05:38.731225 17557 tablet_bootstrap.cc:492] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67: Bootstrap starting.
I20240503 14:05:38.734668 17561 raft_consensus.cc:482] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:38.735188 17561 raft_consensus.cc:504] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.737110 17557 tablet_bootstrap.cc:654] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:38.737247 17561 leader_election.cc:288] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2171b09913424ab3b53d5397987eebcf (127.13.158.66:35177), ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67:34965)
I20240503 14:05:38.749599 17557 tablet_bootstrap.cc:492] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67: No bootstrap required, opened a new log
I20240503 14:05:38.750624 17557 ts_tablet_manager.cc:1367] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.007s
I20240503 14:05:38.751933 17553 tablet_bootstrap.cc:492] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf: No bootstrap required, opened a new log
I20240503 14:05:38.752527 17553 ts_tablet_manager.cc:1367] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf: Time spent bootstrapping tablet: real 0.053s user 0.021s sys 0.008s
I20240503 14:05:38.752914 17508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2032d05f926d4989ba2d00c1ea704f8a" candidate_uuid: "2171b09913424ab3b53d5397987eebcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" is_pre_election: true
I20240503 14:05:38.753857 17508 raft_consensus.cc:2454] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2171b09913424ab3b53d5397987eebcf in term 0.
I20240503 14:05:38.753572 17557 raft_consensus.cc:348] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.754469 17557 raft_consensus.cc:374] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:38.755244 17397 leader_election.cc:302] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2171b09913424ab3b53d5397987eebcf, ab5460d695ad4c1881d5f82b4df8adbf; no voters:
I20240503 14:05:38.755569 17553 raft_consensus.cc:348] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.756243 17560 raft_consensus.cc:2781] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:38.756533 17553 raft_consensus.cc:374] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:38.756665 17560 raft_consensus.cc:482] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:38.757087 17560 raft_consensus.cc:3037] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:38.757164 17557 raft_consensus.cc:729] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ac3c2f11a44d90ba10d615508b5e67, State: Initialized, Role: FOLLOWER
I20240503 14:05:38.758200 17557 consensus_queue.cc:260] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.760536 17557 ts_tablet_manager.cc:1398] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240503 14:05:38.757090 17553 raft_consensus.cc:729] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2171b09913424ab3b53d5397987eebcf, State: Initialized, Role: FOLLOWER
I20240503 14:05:38.762225 17553 consensus_queue.cc:260] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.765256 17560 raft_consensus.cc:504] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:38.768173 17560 leader_election.cc:288] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [CANDIDATE]: Term 1 election: Requested vote from peers ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67:34965), 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65:36197)
I20240503 14:05:38.768561 17359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2032d05f926d4989ba2d00c1ea704f8a" candidate_uuid: "2171b09913424ab3b53d5397987eebcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ac3c2f11a44d90ba10d615508b5e67" is_pre_election: true
I20240503 14:05:38.769426 17359 raft_consensus.cc:2454] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2171b09913424ab3b53d5397987eebcf in term 0.
I20240503 14:05:38.770511 17553 ts_tablet_manager.cc:1398] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf: Time spent starting tablet: real 0.018s user 0.010s sys 0.003s
I20240503 14:05:38.770644 17508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2032d05f926d4989ba2d00c1ea704f8a" candidate_uuid: "2171b09913424ab3b53d5397987eebcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5460d695ad4c1881d5f82b4df8adbf"
I20240503 14:05:38.771293 17508 raft_consensus.cc:3037] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:38.772583 17434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2032d05f926d4989ba2d00c1ea704f8a" candidate_uuid: "30ac3c2f11a44d90ba10d615508b5e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2171b09913424ab3b53d5397987eebcf" is_pre_election: true
I20240503 14:05:38.773434 17434 raft_consensus.cc:2379] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ac3c2f11a44d90ba10d615508b5e67 in current term 1: Already voted for candidate 2171b09913424ab3b53d5397987eebcf in this term.
I20240503 14:05:38.776978 17434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c1915f91a94b4cb25df5f593a14797" candidate_uuid: "30ac3c2f11a44d90ba10d615508b5e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2171b09913424ab3b53d5397987eebcf" is_pre_election: true
I20240503 14:05:38.777652 17434 raft_consensus.cc:2454] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ac3c2f11a44d90ba10d615508b5e67 in term 0.
I20240503 14:05:38.778815 17508 raft_consensus.cc:2454] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2171b09913424ab3b53d5397987eebcf in term 1.
I20240503 14:05:38.779109 17322 leader_election.cc:302] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2171b09913424ab3b53d5397987eebcf, 30ac3c2f11a44d90ba10d615508b5e67; no voters:
I20240503 14:05:38.779613 17359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2032d05f926d4989ba2d00c1ea704f8a" candidate_uuid: "2171b09913424ab3b53d5397987eebcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ac3c2f11a44d90ba10d615508b5e67"
I20240503 14:05:38.781294 17359 raft_consensus.cc:3037] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:38.781512 17561 raft_consensus.cc:2781] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:38.781881 17561 raft_consensus.cc:482] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:38.782222 17561 raft_consensus.cc:3037] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:38.783010 17507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c1915f91a94b4cb25df5f593a14797" candidate_uuid: "30ac3c2f11a44d90ba10d615508b5e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" is_pre_election: true
I20240503 14:05:38.783707 17507 raft_consensus.cc:2454] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ac3c2f11a44d90ba10d615508b5e67 in term 0.
I20240503 14:05:38.784946 17508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2032d05f926d4989ba2d00c1ea704f8a" candidate_uuid: "30ac3c2f11a44d90ba10d615508b5e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" is_pre_election: true
I20240503 14:05:38.780833 17397 leader_election.cc:302] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2171b09913424ab3b53d5397987eebcf, ab5460d695ad4c1881d5f82b4df8adbf; no voters:
I20240503 14:05:38.786549 17508 raft_consensus.cc:2379] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ac3c2f11a44d90ba10d615508b5e67 in current term 1: Already voted for candidate 2171b09913424ab3b53d5397987eebcf in this term.
I20240503 14:05:38.787026 17560 raft_consensus.cc:2781] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:38.787509 17560 raft_consensus.cc:686] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 1 LEADER]: Becoming Leader. State: Replica: 2171b09913424ab3b53d5397987eebcf, State: Running, Role: LEADER
I20240503 14:05:38.787613 17322 leader_election.cc:302] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ac3c2f11a44d90ba10d615508b5e67; no voters: 2171b09913424ab3b53d5397987eebcf, ab5460d695ad4c1881d5f82b4df8adbf
I20240503 14:05:38.788555 17359 raft_consensus.cc:2454] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2171b09913424ab3b53d5397987eebcf in term 1.
I20240503 14:05:38.788364 17560 consensus_queue.cc:237] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:38.791136 17574 raft_consensus.cc:2726] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:38.790814 17561 raft_consensus.cc:504] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.793632 17561 leader_election.cc:288] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [CANDIDATE]: Term 1 election: Requested vote from peers 2171b09913424ab3b53d5397987eebcf (127.13.158.66:35177), ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67:34965)
I20240503 14:05:38.794457 17434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c1915f91a94b4cb25df5f593a14797" candidate_uuid: "30ac3c2f11a44d90ba10d615508b5e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2171b09913424ab3b53d5397987eebcf"
I20240503 14:05:38.795053 17434 raft_consensus.cc:3037] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:38.795281 17508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c1915f91a94b4cb25df5f593a14797" candidate_uuid: "30ac3c2f11a44d90ba10d615508b5e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5460d695ad4c1881d5f82b4df8adbf"
I20240503 14:05:38.795836 17508 raft_consensus.cc:3037] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:38.796589 17254 catalog_manager.cc:5484] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2171b09913424ab3b53d5397987eebcf (127.13.158.66). New cstate: current_term: 1 leader_uuid: "2171b09913424ab3b53d5397987eebcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:38.802321 17434 raft_consensus.cc:2454] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ac3c2f11a44d90ba10d615508b5e67 in term 1.
I20240503 14:05:38.802325 17508 raft_consensus.cc:2454] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ac3c2f11a44d90ba10d615508b5e67 in term 1.
I20240503 14:05:38.803629 17322 leader_election.cc:302] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2171b09913424ab3b53d5397987eebcf, 30ac3c2f11a44d90ba10d615508b5e67; no voters:
I20240503 14:05:38.804980 17561 raft_consensus.cc:2781] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:38.805536 17561 raft_consensus.cc:686] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 LEADER]: Becoming Leader. State: Replica: 30ac3c2f11a44d90ba10d615508b5e67, State: Running, Role: LEADER
I20240503 14:05:38.806586 17561 consensus_queue.cc:237] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.814783 17254 catalog_manager.cc:5484] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65). New cstate: current_term: 1 leader_uuid: "30ac3c2f11a44d90ba10d615508b5e67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:38.862392 17560 raft_consensus.cc:482] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:38.862926 17560 raft_consensus.cc:504] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.864475 17560 leader_election.cc:288] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65:36197), ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67:34965)
I20240503 14:05:38.865514 17508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da79a01e920f448384284e3d77da4293" candidate_uuid: "2171b09913424ab3b53d5397987eebcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" is_pre_election: true
I20240503 14:05:38.865514 17359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da79a01e920f448384284e3d77da4293" candidate_uuid: "2171b09913424ab3b53d5397987eebcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ac3c2f11a44d90ba10d615508b5e67" is_pre_election: true
I20240503 14:05:38.866384 17508 raft_consensus.cc:2454] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2171b09913424ab3b53d5397987eebcf in term 0.
I20240503 14:05:38.866415 17359 raft_consensus.cc:2454] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2171b09913424ab3b53d5397987eebcf in term 0.
I20240503 14:05:38.867467 17397 leader_election.cc:302] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2171b09913424ab3b53d5397987eebcf, ab5460d695ad4c1881d5f82b4df8adbf; no voters:
I20240503 14:05:38.868156 17560 raft_consensus.cc:2781] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:38.868489 17560 raft_consensus.cc:482] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:38.868724 17560 raft_consensus.cc:3037] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:38.873354 17560 raft_consensus.cc:504] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.874939 17560 leader_election.cc:288] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [CANDIDATE]: Term 1 election: Requested vote from peers 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65:36197), ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67:34965)
I20240503 14:05:38.876161 17359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da79a01e920f448384284e3d77da4293" candidate_uuid: "2171b09913424ab3b53d5397987eebcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ac3c2f11a44d90ba10d615508b5e67"
I20240503 14:05:38.876161 17508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da79a01e920f448384284e3d77da4293" candidate_uuid: "2171b09913424ab3b53d5397987eebcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5460d695ad4c1881d5f82b4df8adbf"
I20240503 14:05:38.876988 17359 raft_consensus.cc:3037] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:38.877018 17508 raft_consensus.cc:3037] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:38.884028 17508 raft_consensus.cc:2454] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2171b09913424ab3b53d5397987eebcf in term 1.
I20240503 14:05:38.884028 17359 raft_consensus.cc:2454] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2171b09913424ab3b53d5397987eebcf in term 1.
I20240503 14:05:38.885529 17400 leader_election.cc:302] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2171b09913424ab3b53d5397987eebcf, 30ac3c2f11a44d90ba10d615508b5e67; no voters:
I20240503 14:05:38.886329 17560 raft_consensus.cc:2781] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:38.886807 17560 raft_consensus.cc:686] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 1 LEADER]: Becoming Leader. State: Replica: 2171b09913424ab3b53d5397987eebcf, State: Running, Role: LEADER
I20240503 14:05:38.887615 17560 consensus_queue.cc:237] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } }
I20240503 14:05:38.894872 17254 catalog_manager.cc:5484] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2171b09913424ab3b53d5397987eebcf (127.13.158.66). New cstate: current_term: 1 leader_uuid: "2171b09913424ab3b53d5397987eebcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:38.925221 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:38.932807 17578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:38.933920 17579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:38.934679 17581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:38.935263 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:38.936635 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:38.936858 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:38.937048 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745138937030 us; error 0 us; skew 500 ppm
I20240503 14:05:38.937598 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:38.940292 13945 webserver.cc:460] Webserver started at http://127.13.158.68:40239/ using document root <none> and password file <none>
I20240503 14:05:38.940814 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:38.941026 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:38.941293 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:38.942569 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-3-root/instance:
uuid: "246247aa68024825aab5ffb2fa21e82e"
format_stamp: "Formatted at 2024-05-03 14:05:38 on dist-test-slave-jm3r"
I20240503 14:05:38.947397 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:05:38.950843 17586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:38.951601 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:05:38.951922 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-3-root
uuid: "246247aa68024825aab5ffb2fa21e82e"
format_stamp: "Formatted at 2024-05-03 14:05:38 on dist-test-slave-jm3r"
I20240503 14:05:38.952222 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:38.961266 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:38.962646 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:38.964802 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:38.968155 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:38.968390 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:38.968653 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:38.968837 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:39.001925 17309 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:39.012758 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.68:43341
I20240503 14:05:39.013012 17649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.68:43341 every 8 connection(s)
I20240503 14:05:39.018853 17434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2032d05f926d4989ba2d00c1ea704f8a"
dest_uuid: "2171b09913424ab3b53d5397987eebcf"
mode: GRACEFUL
new_leader_uuid: "ab5460d695ad4c1881d5f82b4df8adbf"
from {username='slave'} at 127.0.0.1:56372
I20240503 14:05:39.019816 17434 raft_consensus.cc:595] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 1 LEADER]: Received request to transfer leadership to TS ab5460d695ad4c1881d5f82b4df8adbf
I20240503 14:05:39.021595 17309 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 14:05:39.022166 17309 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:39.033838 17650 heartbeater.cc:347] Connected to a master server at 127.13.158.126:46229
I20240503 14:05:39.034230 17650 heartbeater.cc:464] Registering TS with master...
I20240503 14:05:39.035005 17650 heartbeater.cc:511] Master 127.13.158.126:46229 requested a full tablet report, sending...
I20240503 14:05:39.036846 17254 ts_manager.cc:194] Registered new tserver with Master: 246247aa68024825aab5ffb2fa21e82e (127.13.158.68:43341)
I20240503 14:05:39.038290 17254 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58620
I20240503 14:05:39.062673 17560 consensus_queue.cc:1035] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:05:39.069492 17573 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240503 14:05:39.069823 17573 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240503 14:05:39.071324 17573 raft_consensus.cc:979] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf: : Instructing follower ab5460d695ad4c1881d5f82b4df8adbf to start an election
I20240503 14:05:39.071811 17573 raft_consensus.cc:1067] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 1 LEADER]: Signalling peer ab5460d695ad4c1881d5f82b4df8adbf to start an election
I20240503 14:05:39.073576 17508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2032d05f926d4989ba2d00c1ea704f8a"
dest_uuid: "ab5460d695ad4c1881d5f82b4df8adbf"
from {username='slave'} at 127.0.0.1:54866
I20240503 14:05:39.074167 17508 raft_consensus.cc:482] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:05:39.074523 17508 raft_consensus.cc:3037] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:05:39.079504 17508 raft_consensus.cc:504] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:39.081413 17508 leader_election.cc:288] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [CANDIDATE]: Term 2 election: Requested vote from peers 2171b09913424ab3b53d5397987eebcf (127.13.158.66:35177), 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65:36197)
I20240503 14:05:39.086347 17573 consensus_queue.cc:1035] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:39.096163 17434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2032d05f926d4989ba2d00c1ea704f8a" candidate_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2171b09913424ab3b53d5397987eebcf"
I20240503 14:05:39.096887 17434 raft_consensus.cc:3032] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:05:39.097170 17434 raft_consensus.cc:729] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2171b09913424ab3b53d5397987eebcf, State: Running, Role: LEADER
I20240503 14:05:39.097827 17434 consensus_queue.cc:260] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:39.098886 17434 raft_consensus.cc:3037] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:05:39.099546 17359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2032d05f926d4989ba2d00c1ea704f8a" candidate_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30ac3c2f11a44d90ba10d615508b5e67"
I20240503 14:05:39.100203 17359 raft_consensus.cc:3037] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:05:39.103775 17434 raft_consensus.cc:2454] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5460d695ad4c1881d5f82b4df8adbf in term 2.
I20240503 14:05:39.104974 17471 leader_election.cc:302] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2171b09913424ab3b53d5397987eebcf, ab5460d695ad4c1881d5f82b4df8adbf; no voters:
I20240503 14:05:39.105090 17359 raft_consensus.cc:2454] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5460d695ad4c1881d5f82b4df8adbf in term 2.
I20240503 14:05:39.106024 17559 raft_consensus.cc:2781] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:05:39.107580 17559 raft_consensus.cc:686] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 2 LEADER]: Becoming Leader. State: Replica: ab5460d695ad4c1881d5f82b4df8adbf, State: Running, Role: LEADER
I20240503 14:05:39.108479 17559 consensus_queue.cc:237] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } }
I20240503 14:05:39.114931 17254 catalog_manager.cc:5484] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf reported cstate change: term changed from 1 to 2, leader changed from 2171b09913424ab3b53d5397987eebcf (127.13.158.66) to ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67). New cstate: current_term: 2 leader_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:39.122810 17573 consensus_queue.cc:1035] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:39.123769 17561 consensus_queue.cc:1035] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:39.146114 17561 consensus_queue.cc:1035] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:39.245229 17573 consensus_queue.cc:1035] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:39.410518 17359 raft_consensus.cc:1261] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 2 FOLLOWER]: Refusing update from remote peer ab5460d695ad4c1881d5f82b4df8adbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:05:39.411932 17559 consensus_queue.cc:1035] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:39.442935 17434 raft_consensus.cc:1261] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 2 FOLLOWER]: Refusing update from remote peer ab5460d695ad4c1881d5f82b4df8adbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:05:39.444087 17559 consensus_queue.cc:1035] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:40.023304 17507 consensus_queue.cc:237] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } }
I20240503 14:05:40.028824 17434 raft_consensus.cc:1261] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 2 FOLLOWER]: Refusing update from remote peer ab5460d695ad4c1881d5f82b4df8adbf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:05:40.029022 17359 raft_consensus.cc:1261] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 2 FOLLOWER]: Refusing update from remote peer ab5460d695ad4c1881d5f82b4df8adbf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:05:40.030750 17660 consensus_queue.cc:1035] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:40.031754 17668 consensus_queue.cc:1035] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:40.040920 17650 heartbeater.cc:503] Master 127.13.158.126:46229 was elected leader, sending a full tablet report...
W20240503 14:05:40.042914 17472 consensus_peers.cc:485] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf -> Peer 246247aa68024825aab5ffb2fa21e82e (127.13.158.68:43341): Couldn't send request to peer 246247aa68024825aab5ffb2fa21e82e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2032d05f926d4989ba2d00c1ea704f8a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:40.040792 17434 raft_consensus.cc:2932] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } } }
I20240503 14:05:40.043917 17359 raft_consensus.cc:2932] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } } }
I20240503 14:05:40.055544 17254 catalog_manager.cc:5484] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf reported cstate change: config changed from index -1 to 3, NON_VOTER 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) added. New cstate: current_term: 2 leader_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } } }
I20240503 14:05:40.269356 17660 raft_consensus.cc:2932] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } } }
I20240503 14:05:40.290287 17434 consensus_queue.cc:237] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } }
I20240503 14:05:40.296875 17359 raft_consensus.cc:1261] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Refusing update from remote peer 2171b09913424ab3b53d5397987eebcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:40.297116 17507 raft_consensus.cc:1261] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Refusing update from remote peer 2171b09913424ab3b53d5397987eebcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:40.299079 17560 consensus_queue.cc:1035] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:05:40.300244 17678 consensus_queue.cc:1035] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:05:40.309823 17398 consensus_peers.cc:485] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf -> Peer 246247aa68024825aab5ffb2fa21e82e (127.13.158.68:43341): Couldn't send request to peer 246247aa68024825aab5ffb2fa21e82e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da79a01e920f448384284e3d77da4293. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:40.313114 17507 raft_consensus.cc:2932] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } } }
I20240503 14:05:40.314260 17359 raft_consensus.cc:2932] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } } }
I20240503 14:05:40.323938 17256 catalog_manager.cc:5484] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf reported cstate change: config changed from index -1 to 2, NON_VOTER 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) added. New cstate: current_term: 1 leader_uuid: "2171b09913424ab3b53d5397987eebcf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } } }
I20240503 14:05:40.392531 17683 ts_tablet_manager.cc:902] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e: Initiating tablet copy from peer ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67:34965)
I20240503 14:05:40.396096 17683 tablet_copy_client.cc:308] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e: tablet copy: Beginning tablet copy session from remote peer at address 127.13.158.67:34965
I20240503 14:05:40.406047 17518 tablet_copy_service.cc:140] P ab5460d695ad4c1881d5f82b4df8adbf: Received BeginTabletCopySession request for tablet 2032d05f926d4989ba2d00c1ea704f8a from peer 246247aa68024825aab5ffb2fa21e82e ({username='slave'} at 127.0.0.1:37886)
I20240503 14:05:40.406579 17518 tablet_copy_service.cc:161] P ab5460d695ad4c1881d5f82b4df8adbf: Beginning new tablet copy session on tablet 2032d05f926d4989ba2d00c1ea704f8a from peer 246247aa68024825aab5ffb2fa21e82e at {username='slave'} at 127.0.0.1:37886: session id = 246247aa68024825aab5ffb2fa21e82e-2032d05f926d4989ba2d00c1ea704f8a
I20240503 14:05:40.412906 17518 tablet_copy_source_session.cc:205] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:40.415496 17683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2032d05f926d4989ba2d00c1ea704f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:40.426920 17683 tablet_copy_client.cc:783] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:40.427457 17683 tablet_copy_client.cc:647] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:40.430845 17683 tablet_copy_client.cc:523] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:40.436911 17683 tablet_bootstrap.cc:492] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e: Bootstrap starting.
I20240503 14:05:40.453097 17683 tablet_bootstrap.cc:492] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:40.453882 17683 tablet_bootstrap.cc:492] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e: Bootstrap complete.
I20240503 14:05:40.454455 17683 ts_tablet_manager.cc:1367] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240503 14:05:40.456383 17683 raft_consensus.cc:348] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } }
I20240503 14:05:40.456998 17683 raft_consensus.cc:729] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 246247aa68024825aab5ffb2fa21e82e, State: Initialized, Role: LEARNER
I20240503 14:05:40.457535 17683 consensus_queue.cc:260] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } }
I20240503 14:05:40.459862 17683 ts_tablet_manager.cc:1398] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240503 14:05:40.461599 17518 tablet_copy_service.cc:342] P ab5460d695ad4c1881d5f82b4df8adbf: Request end of tablet copy session 246247aa68024825aab5ffb2fa21e82e-2032d05f926d4989ba2d00c1ea704f8a received from {username='slave'} at 127.0.0.1:37886
I20240503 14:05:40.462014 17518 tablet_copy_service.cc:433] P ab5460d695ad4c1881d5f82b4df8adbf: ending tablet copy session 246247aa68024825aab5ffb2fa21e82e-2032d05f926d4989ba2d00c1ea704f8a on tablet 2032d05f926d4989ba2d00c1ea704f8a with peer 246247aa68024825aab5ffb2fa21e82e
I20240503 14:05:40.543751 17560 raft_consensus.cc:2932] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } } }
I20240503 14:05:40.587823 17683 ts_tablet_manager.cc:902] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e: Initiating tablet copy from peer 2171b09913424ab3b53d5397987eebcf (127.13.158.66:35177)
I20240503 14:05:40.592046 17683 tablet_copy_client.cc:308] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e: tablet copy: Beginning tablet copy session from remote peer at address 127.13.158.66:35177
I20240503 14:05:40.600622 17444 tablet_copy_service.cc:140] P 2171b09913424ab3b53d5397987eebcf: Received BeginTabletCopySession request for tablet da79a01e920f448384284e3d77da4293 from peer 246247aa68024825aab5ffb2fa21e82e ({username='slave'} at 127.0.0.1:56386)
I20240503 14:05:40.601228 17444 tablet_copy_service.cc:161] P 2171b09913424ab3b53d5397987eebcf: Beginning new tablet copy session on tablet da79a01e920f448384284e3d77da4293 from peer 246247aa68024825aab5ffb2fa21e82e at {username='slave'} at 127.0.0.1:56386: session id = 246247aa68024825aab5ffb2fa21e82e-da79a01e920f448384284e3d77da4293
I20240503 14:05:40.607936 17444 tablet_copy_source_session.cc:205] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:40.610706 17683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da79a01e920f448384284e3d77da4293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:40.624190 17683 tablet_copy_client.cc:783] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:40.624744 17683 tablet_copy_client.cc:647] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:40.628284 17683 tablet_copy_client.cc:523] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:40.636953 17683 tablet_bootstrap.cc:492] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e: Bootstrap starting.
I20240503 14:05:40.656051 17683 tablet_bootstrap.cc:492] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:40.656924 17683 tablet_bootstrap.cc:492] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e: Bootstrap complete.
I20240503 14:05:40.657464 17683 ts_tablet_manager.cc:1367] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20240503 14:05:40.659979 17683 raft_consensus.cc:348] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } }
I20240503 14:05:40.660584 17683 raft_consensus.cc:729] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 246247aa68024825aab5ffb2fa21e82e, State: Initialized, Role: LEARNER
I20240503 14:05:40.661109 17683 consensus_queue.cc:260] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: true } }
I20240503 14:05:40.663342 17683 ts_tablet_manager.cc:1398] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240503 14:05:40.666198 17444 tablet_copy_service.cc:342] P 2171b09913424ab3b53d5397987eebcf: Request end of tablet copy session 246247aa68024825aab5ffb2fa21e82e-da79a01e920f448384284e3d77da4293 received from {username='slave'} at 127.0.0.1:56386
I20240503 14:05:40.666648 17444 tablet_copy_service.cc:433] P 2171b09913424ab3b53d5397987eebcf: ending tablet copy session 246247aa68024825aab5ffb2fa21e82e-da79a01e920f448384284e3d77da4293 on tablet da79a01e920f448384284e3d77da4293 with peer 246247aa68024825aab5ffb2fa21e82e
I20240503 14:05:40.763557 17624 raft_consensus.cc:1203] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 14:05:40.833951 17624 raft_consensus.cc:1203] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:05:41.023217 17309 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:41.026126 17309 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:41.027257 17309 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:41.238034 17560 raft_consensus.cc:1050] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf: attempting to promote NON_VOTER 246247aa68024825aab5ffb2fa21e82e to VOTER
I20240503 14:05:41.240387 17560 consensus_queue.cc:237] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } }
I20240503 14:05:41.245712 17624 raft_consensus.cc:1261] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e [term 1 LEARNER]: Refusing update from remote peer 2171b09913424ab3b53d5397987eebcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:41.246769 17508 raft_consensus.cc:1261] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Refusing update from remote peer 2171b09913424ab3b53d5397987eebcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:41.246871 17359 raft_consensus.cc:1261] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Refusing update from remote peer 2171b09913424ab3b53d5397987eebcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:41.247618 17678 consensus_queue.cc:1035] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:41.248649 17560 consensus_queue.cc:1035] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:41.249400 17691 consensus_queue.cc:1035] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:41.260556 17359 raft_consensus.cc:2932] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.261361 17624 raft_consensus.cc:2932] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.274338 17256 catalog_manager.cc:5484] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 reported cstate change: config changed from index 2 to 3, 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2171b09913424ab3b53d5397987eebcf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.294697 17676 raft_consensus.cc:1050] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf: attempting to promote NON_VOTER 246247aa68024825aab5ffb2fa21e82e to VOTER
I20240503 14:05:41.296664 17676 consensus_queue.cc:237] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } }
I20240503 14:05:41.302793 17624 raft_consensus.cc:1261] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e [term 2 LEARNER]: Refusing update from remote peer ab5460d695ad4c1881d5f82b4df8adbf: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:05:41.302793 17359 raft_consensus.cc:1261] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 2 FOLLOWER]: Refusing update from remote peer ab5460d695ad4c1881d5f82b4df8adbf: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:05:41.304646 17660 consensus_queue.cc:1035] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:41.305397 17434 raft_consensus.cc:1261] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 2 FOLLOWER]: Refusing update from remote peer ab5460d695ad4c1881d5f82b4df8adbf: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:05:41.305752 17701 consensus_queue.cc:1035] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:41.307168 17676 consensus_queue.cc:1035] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:41.322635 17433 raft_consensus.cc:2932] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.321686 17359 raft_consensus.cc:2932] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.335266 17255 catalog_manager.cc:5484] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf reported cstate change: config changed from index 3 to 4, 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.528977 17508 raft_consensus.cc:2932] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.530663 17701 raft_consensus.cc:2932] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.532351 17678 raft_consensus.cc:2932] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.542804 17624 raft_consensus.cc:2932] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 246247aa68024825aab5ffb2fa21e82e (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } attrs { replace: true } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.647665 17508 consensus_queue.cc:237] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } }
I20240503 14:05:41.654763 17434 raft_consensus.cc:1261] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 2 FOLLOWER]: Refusing update from remote peer ab5460d695ad4c1881d5f82b4df8adbf: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:05:41.654860 17624 raft_consensus.cc:1261] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e [term 2 FOLLOWER]: Refusing update from remote peer ab5460d695ad4c1881d5f82b4df8adbf: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:05:41.656526 17433 consensus_queue.cc:237] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } }
I20240503 14:05:41.656958 17706 consensus_queue.cc:1035] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:05:41.657927 17676 consensus_queue.cc:1035] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:05:41.663216 17359 raft_consensus.cc:1261] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Refusing update from remote peer 2171b09913424ab3b53d5397987eebcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:41.663718 17624 raft_consensus.cc:1261] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e [term 1 FOLLOWER]: Refusing update from remote peer 2171b09913424ab3b53d5397987eebcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:41.666769 17691 consensus_queue.cc:1035] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:41.672319 17691 consensus_queue.cc:1035] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:41.673364 17433 raft_consensus.cc:2932] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.674536 17624 raft_consensus.cc:2932] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.677376 17359 raft_consensus.cc:2932] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.682873 17623 raft_consensus.cc:2932] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.690958 17256 catalog_manager.cc:5484] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e reported cstate change: config changed from index 4 to 5, VOTER 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65) evicted. New cstate: current_term: 2 leader_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.694041 17254 catalog_manager.cc:5484] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 reported cstate change: config changed from index 3 to 4, VOTER ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67) evicted. New cstate: current_term: 1 leader_uuid: "2171b09913424ab3b53d5397987eebcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.845633 17496 tablet_service.cc:1514] Processing DeleteTablet for tablet da79a01e920f448384284e3d77da4293 with delete_type TABLET_DATA_TOMBSTONED (TS ab5460d695ad4c1881d5f82b4df8adbf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54860
I20240503 14:05:41.847668 17707 tablet_replica.cc:330] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf: stopping tablet replica
I20240503 14:05:41.848327 17707 raft_consensus.cc:2229] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:41.848927 17707 raft_consensus.cc:2258] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:41.851720 17707 ts_tablet_manager.cc:1875] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:41.862594 17707 ts_tablet_manager.cc:1888] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:05:41.862923 17707 log.cc:1196] T da79a01e920f448384284e3d77da4293 P ab5460d695ad4c1881d5f82b4df8adbf: Deleting WAL directory at /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-2-root/wals/da79a01e920f448384284e3d77da4293
I20240503 14:05:41.864718 17240 catalog_manager.cc:4830] TS ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67:34965): tablet da79a01e920f448384284e3d77da4293 (table test-workload [id=13b3d0790bb342d5bbe9211b1f9acf0d]) successfully deleted
I20240503 14:05:41.922927 17691 raft_consensus.cc:2932] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5460d695ad4c1881d5f82b4df8adbf (127.13.158.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "30ac3c2f11a44d90ba10d615508b5e67" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 36197 } } peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:41.929566 17240 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet da79a01e920f448384284e3d77da4293 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:42.012867 17347 tablet_service.cc:1514] Processing DeleteTablet for tablet 2032d05f926d4989ba2d00c1ea704f8a with delete_type TABLET_DATA_TOMBSTONED (TS 30ac3c2f11a44d90ba10d615508b5e67 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44084
I20240503 14:05:42.014842 17708 tablet_replica.cc:330] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67: stopping tablet replica
I20240503 14:05:42.017809 17708 raft_consensus.cc:2229] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:42.018649 17708 raft_consensus.cc:2258] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:42.021953 17708 ts_tablet_manager.cc:1875] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:42.036111 17708 ts_tablet_manager.cc:1888] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 14:05:42.036507 17708 log.cc:1196] T 2032d05f926d4989ba2d00c1ea704f8a P 30ac3c2f11a44d90ba10d615508b5e67: Deleting WAL directory at /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714745036514908-13945-0/minicluster-data/ts-0-root/wals/2032d05f926d4989ba2d00c1ea704f8a
I20240503 14:05:42.038084 17243 catalog_manager.cc:4830] TS 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65:36197): tablet 2032d05f926d4989ba2d00c1ea704f8a (table test-workload [id=13b3d0790bb342d5bbe9211b1f9acf0d]) successfully deleted
I20240503 14:05:42.369460 17676 raft_consensus.cc:2932] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2171b09913424ab3b53d5397987eebcf" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 35177 } } peers { permanent_uuid: "ab5460d695ad4c1881d5f82b4df8adbf" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34965 } } peers { permanent_uuid: "246247aa68024825aab5ffb2fa21e82e" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 43341 } attrs { promote: false } } }
I20240503 14:05:42.376242 17240 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2032d05f926d4989ba2d00c1ea704f8a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:43.028264 17309 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:43.030779 17309 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:43.031145 17309 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:45.032012 17309 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:45.034708 17309 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:45.035099 17309 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:47.036047 17309 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:47.039489 17309 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:47.039914 17309 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:49.041085 17309 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:49.043946 17309 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:49.044358 17309 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:49.090529 13945 tablet_server.cc:178] TabletServer@127.13.158.65:0 shutting down...
I20240503 14:05:49.119256 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:49.119983 13945 tablet_replica.cc:330] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67: stopping tablet replica
I20240503 14:05:49.120709 13945 raft_consensus.cc:2229] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:49.122107 13945 raft_consensus.cc:2258] T da79a01e920f448384284e3d77da4293 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:49.124629 13945 tablet_replica.cc:330] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67: stopping tablet replica
I20240503 14:05:49.125289 13945 raft_consensus.cc:2229] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:05:49.126272 13945 raft_consensus.cc:2258] T a3c1915f91a94b4cb25df5f593a14797 P 30ac3c2f11a44d90ba10d615508b5e67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:49.151249 13945 tablet_server.cc:194] TabletServer@127.13.158.65:0 shutdown complete.
W20240503 14:05:49.166241 17400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.158.65:36197: connect: Connection refused (error 111) [suppressed 5 similar messages]
I20240503 14:05:49.168918 13945 tablet_server.cc:178] TabletServer@127.13.158.66:0 shutting down...
W20240503 14:05:49.175544 17400 consensus_peers.cc:485] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf -> Peer 30ac3c2f11a44d90ba10d615508b5e67 (127.13.158.65:36197): Couldn't send request to peer 30ac3c2f11a44d90ba10d615508b5e67. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:36197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:49.196530 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:49.197259 13945 tablet_replica.cc:330] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf: stopping tablet replica
I20240503 14:05:49.198153 13945 raft_consensus.cc:2229] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:05:49.199108 13945 raft_consensus.cc:2258] T da79a01e920f448384284e3d77da4293 P 2171b09913424ab3b53d5397987eebcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:49.201160 13945 tablet_replica.cc:330] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf: stopping tablet replica
I20240503 14:05:49.201606 13945 raft_consensus.cc:2229] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:49.202025 13945 raft_consensus.cc:2258] T a3c1915f91a94b4cb25df5f593a14797 P 2171b09913424ab3b53d5397987eebcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:49.204169 13945 tablet_replica.cc:330] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf: stopping tablet replica
I20240503 14:05:49.204627 13945 raft_consensus.cc:2229] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:49.205191 13945 raft_consensus.cc:2258] T 2032d05f926d4989ba2d00c1ea704f8a P 2171b09913424ab3b53d5397987eebcf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:49.228968 13945 tablet_server.cc:194] TabletServer@127.13.158.66:0 shutdown complete.
I20240503 14:05:49.244978 13945 tablet_server.cc:178] TabletServer@127.13.158.67:0 shutting down...
I20240503 14:05:49.282945 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:49.283799 13945 tablet_replica.cc:330] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf: stopping tablet replica
I20240503 14:05:49.284570 13945 raft_consensus.cc:2229] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:49.285120 13945 raft_consensus.cc:2258] T a3c1915f91a94b4cb25df5f593a14797 P ab5460d695ad4c1881d5f82b4df8adbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:49.287798 13945 tablet_replica.cc:330] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf: stopping tablet replica
I20240503 14:05:49.288376 13945 raft_consensus.cc:2229] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 2 LEADER]: Raft consensus shutting down.
I20240503 14:05:49.289548 13945 raft_consensus.cc:2258] T 2032d05f926d4989ba2d00c1ea704f8a P ab5460d695ad4c1881d5f82b4df8adbf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:49.316988 13945 tablet_server.cc:194] TabletServer@127.13.158.67:0 shutdown complete.
I20240503 14:05:49.330541 13945 tablet_server.cc:178] TabletServer@127.13.158.68:0 shutting down...
I20240503 14:05:49.352689 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:49.353381 13945 tablet_replica.cc:330] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e: stopping tablet replica
I20240503 14:05:49.353974 13945 raft_consensus.cc:2229] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:49.354769 13945 raft_consensus.cc:2258] T da79a01e920f448384284e3d77da4293 P 246247aa68024825aab5ffb2fa21e82e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:49.356783 13945 tablet_replica.cc:330] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e: stopping tablet replica
I20240503 14:05:49.357270 13945 raft_consensus.cc:2229] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:49.357795 13945 raft_consensus.cc:2258] T 2032d05f926d4989ba2d00c1ea704f8a P 246247aa68024825aab5ffb2fa21e82e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:49.377878 13945 tablet_server.cc:194] TabletServer@127.13.158.68:0 shutdown complete.
I20240503 14:05:49.389236 13945 master.cc:537] Master@127.13.158.126:46229 shutting down...
I20240503 14:05:49.406244 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:05:49.406926 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:49.407291 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 304b2a18064d4920804af80fc1555430: stopping tablet replica
I20240503 14:05:49.424875 13945 master.cc:552] Master@127.13.158.126:46229 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (12614 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20240503 14:05:49.461742 13945 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.13.158.126:39803
I20240503 14:05:49.462848 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:49.469449 17713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:49.470065 17714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:49.470945 17716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:49.471725 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:49.472522 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:49.472704 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:49.472847 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745149472829 us; error 0 us; skew 500 ppm
I20240503 14:05:49.473338 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:49.475951 13945 webserver.cc:460] Webserver started at http://127.13.158.126:33801/ using document root <none> and password file <none>
I20240503 14:05:49.476413 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:49.476576 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:49.476840 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:49.478032 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/master-0-root/instance:
uuid: "ee38cd4a3e2b4f6999795de896d76f3e"
format_stamp: "Formatted at 2024-05-03 14:05:49 on dist-test-slave-jm3r"
I20240503 14:05:49.482784 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20240503 14:05:49.486095 17721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:49.486882 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:05:49.487151 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/master-0-root
uuid: "ee38cd4a3e2b4f6999795de896d76f3e"
format_stamp: "Formatted at 2024-05-03 14:05:49 on dist-test-slave-jm3r"
I20240503 14:05:49.487442 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:49.523687 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:49.524981 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:49.562177 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.126:39803
I20240503 14:05:49.562312 17772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.126:39803 every 8 connection(s)
I20240503 14:05:49.566071 17773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:49.577306 17773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e: Bootstrap starting.
I20240503 14:05:49.581462 17773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:49.585546 17773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e: No bootstrap required, opened a new log
I20240503 14:05:49.587589 17773 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee38cd4a3e2b4f6999795de896d76f3e" member_type: VOTER }
I20240503 14:05:49.588143 17773 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:49.588379 17773 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee38cd4a3e2b4f6999795de896d76f3e, State: Initialized, Role: FOLLOWER
I20240503 14:05:49.588877 17773 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee38cd4a3e2b4f6999795de896d76f3e" member_type: VOTER }
I20240503 14:05:49.589381 17773 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:05:49.589576 17773 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:05:49.589795 17773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:49.594172 17773 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee38cd4a3e2b4f6999795de896d76f3e" member_type: VOTER }
I20240503 14:05:49.594678 17773 leader_election.cc:302] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee38cd4a3e2b4f6999795de896d76f3e; no voters:
I20240503 14:05:49.595724 17773 leader_election.cc:288] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:05:49.596101 17776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:49.597600 17776 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [term 1 LEADER]: Becoming Leader. State: Replica: ee38cd4a3e2b4f6999795de896d76f3e, State: Running, Role: LEADER
I20240503 14:05:49.598229 17776 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee38cd4a3e2b4f6999795de896d76f3e" member_type: VOTER }
I20240503 14:05:49.599040 17773 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:05:49.603224 17777 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee38cd4a3e2b4f6999795de896d76f3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee38cd4a3e2b4f6999795de896d76f3e" member_type: VOTER } }
I20240503 14:05:49.603492 17778 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee38cd4a3e2b4f6999795de896d76f3e. Latest consensus state: current_term: 1 leader_uuid: "ee38cd4a3e2b4f6999795de896d76f3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee38cd4a3e2b4f6999795de896d76f3e" member_type: VOTER } }
I20240503 14:05:49.604132 17778 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [sys.catalog]: This master's current role is: LEADER
I20240503 14:05:49.603935 17777 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [sys.catalog]: This master's current role is: LEADER
I20240503 14:05:49.607183 17783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:05:49.614148 17783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:05:49.615137 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:05:49.623339 17783 catalog_manager.cc:1347] Generated new cluster ID: 66263c00129b4f9e9a63dec7980510b3
I20240503 14:05:49.623646 17783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:05:49.644754 17783 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:05:49.646111 17783 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:05:49.657078 17783 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e: Generated new TSK 0
I20240503 14:05:49.657711 17783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:05:49.682065 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:49.688871 17794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:49.689685 17795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:49.690682 17797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:49.691459 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:49.692255 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:49.692440 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:49.692587 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745149692565 us; error 0 us; skew 500 ppm
I20240503 14:05:49.693109 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:49.695472 13945 webserver.cc:460] Webserver started at http://127.13.158.65:40113/ using document root <none> and password file <none>
I20240503 14:05:49.695914 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:49.696076 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:49.696339 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:49.697423 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-0-root/instance:
uuid: "8db7b8c1d60b47b8b8737489cbd0ac93"
format_stamp: "Formatted at 2024-05-03 14:05:49 on dist-test-slave-jm3r"
I20240503 14:05:49.701999 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:05:49.705276 17802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:49.706037 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:05:49.706317 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-0-root
uuid: "8db7b8c1d60b47b8b8737489cbd0ac93"
format_stamp: "Formatted at 2024-05-03 14:05:49 on dist-test-slave-jm3r"
I20240503 14:05:49.706636 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:49.727151 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:49.728253 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:49.729681 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:49.732118 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:49.732337 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:49.732558 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:49.732722 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:49.772358 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.65:38369
I20240503 14:05:49.772469 17864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.65:38369 every 8 connection(s)
I20240503 14:05:49.776832 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:49.786002 17869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:49.787055 17870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:49.789686 17865 heartbeater.cc:347] Connected to a master server at 127.13.158.126:39803
I20240503 14:05:49.790149 17865 heartbeater.cc:464] Registering TS with master...
W20240503 14:05:49.790462 17872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:49.790997 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:49.791206 17865 heartbeater.cc:511] Master 127.13.158.126:39803 requested a full tablet report, sending...
I20240503 14:05:49.792359 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:49.792757 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:49.793076 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745149793037 us; error 0 us; skew 500 ppm
I20240503 14:05:49.793696 17738 ts_manager.cc:194] Registered new tserver with Master: 8db7b8c1d60b47b8b8737489cbd0ac93 (127.13.158.65:38369)
I20240503 14:05:49.793821 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:49.795892 17738 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58684
I20240503 14:05:49.796658 13945 webserver.cc:460] Webserver started at http://127.13.158.66:34959/ using document root <none> and password file <none>
I20240503 14:05:49.797233 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:49.797415 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:49.797672 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:49.798982 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-1-root/instance:
uuid: "25852080845240a8b99d8ba16719b0da"
format_stamp: "Formatted at 2024-05-03 14:05:49 on dist-test-slave-jm3r"
I20240503 14:05:49.803620 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:05:49.807019 17877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:49.807869 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240503 14:05:49.808213 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-1-root
uuid: "25852080845240a8b99d8ba16719b0da"
format_stamp: "Formatted at 2024-05-03 14:05:49 on dist-test-slave-jm3r"
I20240503 14:05:49.808581 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:49.833742 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:49.834993 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:49.836503 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:49.838930 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:49.839156 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:49.839373 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:49.839519 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:49.879612 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.66:44789
I20240503 14:05:49.879696 17939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.66:44789 every 8 connection(s)
I20240503 14:05:49.884269 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:49.891877 17943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:49.892500 17944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:49.893950 17946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:49.895376 13945 server_base.cc:1034] running on GCE node
I20240503 14:05:49.895843 17940 heartbeater.cc:347] Connected to a master server at 127.13.158.126:39803
I20240503 14:05:49.896188 17940 heartbeater.cc:464] Registering TS with master...
I20240503 14:05:49.896294 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:49.896541 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:49.896698 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745149896680 us; error 0 us; skew 500 ppm
I20240503 14:05:49.896900 17940 heartbeater.cc:511] Master 127.13.158.126:39803 requested a full tablet report, sending...
I20240503 14:05:49.897264 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:49.898960 17738 ts_manager.cc:194] Registered new tserver with Master: 25852080845240a8b99d8ba16719b0da (127.13.158.66:44789)
I20240503 14:05:49.900090 13945 webserver.cc:460] Webserver started at http://127.13.158.67:42153/ using document root <none> and password file <none>
I20240503 14:05:49.900553 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:49.900561 17738 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58688
I20240503 14:05:49.900789 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:49.901147 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:49.902629 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-2-root/instance:
uuid: "eceaf08d0aef4b71b49636ba75543e58"
format_stamp: "Formatted at 2024-05-03 14:05:49 on dist-test-slave-jm3r"
I20240503 14:05:49.907096 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:05:49.911118 17951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:49.911871 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 14:05:49.912165 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-2-root
uuid: "eceaf08d0aef4b71b49636ba75543e58"
format_stamp: "Formatted at 2024-05-03 14:05:49 on dist-test-slave-jm3r"
I20240503 14:05:49.912444 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714745036514908-13945-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:49.929111 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:49.930351 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:49.931910 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:49.934252 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:49.934507 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240503 14:05:49.934741 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:49.934892 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:49.973717 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.67:39217
I20240503 14:05:49.973807 18013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.67:39217 every 8 connection(s)
I20240503 14:05:49.986729 18014 heartbeater.cc:347] Connected to a master server at 127.13.158.126:39803
I20240503 14:05:49.987098 18014 heartbeater.cc:464] Registering TS with master...
I20240503 14:05:49.987819 18014 heartbeater.cc:511] Master 127.13.158.126:39803 requested a full tablet report, sending...
I20240503 14:05:49.989701 17738 ts_manager.cc:194] Registered new tserver with Master: eceaf08d0aef4b71b49636ba75543e58 (127.13.158.67:39217)
I20240503 14:05:49.990463 13945 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013613277s
I20240503 14:05:49.991343 17738 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58700
I20240503 14:05:50.798735 17865 heartbeater.cc:503] Master 127.13.158.126:39803 was elected leader, sending a full tablet report...
I20240503 14:05:50.903199 17940 heartbeater.cc:503] Master 127.13.158.126:39803 was elected leader, sending a full tablet report...
I20240503 14:05:50.994169 18014 heartbeater.cc:503] Master 127.13.158.126:39803 was elected leader, sending a full tablet report...
I20240503 14:05:51.023309 13945 test_util.cc:268] Using random seed: 932931766
I20240503 14:05:51.044591 17738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58708:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:05:51.046550 17738 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:05:51.081087 17830 tablet_service.cc:1467] Processing CreateTablet for tablet bd354b08a3fa47b0a039244adad9465e (DEFAULT_TABLE table=test-workload [id=2186d0e4d88e4e7395ef32c2f011961f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:05:51.082388 17830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd354b08a3fa47b0a039244adad9465e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:51.086812 17979 tablet_service.cc:1467] Processing CreateTablet for tablet bd354b08a3fa47b0a039244adad9465e (DEFAULT_TABLE table=test-workload [id=2186d0e4d88e4e7395ef32c2f011961f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:05:51.088088 17979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd354b08a3fa47b0a039244adad9465e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:51.088887 17905 tablet_service.cc:1467] Processing CreateTablet for tablet bd354b08a3fa47b0a039244adad9465e (DEFAULT_TABLE table=test-workload [id=2186d0e4d88e4e7395ef32c2f011961f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:05:51.090082 17905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd354b08a3fa47b0a039244adad9465e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:51.099856 18034 tablet_bootstrap.cc:492] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93: Bootstrap starting.
I20240503 14:05:51.104748 18035 tablet_bootstrap.cc:492] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58: Bootstrap starting.
I20240503 14:05:51.108281 18034 tablet_bootstrap.cc:654] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:51.110706 18035 tablet_bootstrap.cc:654] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:51.111616 18036 tablet_bootstrap.cc:492] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da: Bootstrap starting.
I20240503 14:05:51.114698 18034 tablet_bootstrap.cc:492] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93: No bootstrap required, opened a new log
I20240503 14:05:51.115319 18034 ts_tablet_manager.cc:1367] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20240503 14:05:51.116880 18035 tablet_bootstrap.cc:492] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58: No bootstrap required, opened a new log
I20240503 14:05:51.117447 18035 ts_tablet_manager.cc:1367] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20240503 14:05:51.118244 18034 raft_consensus.cc:348] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } }
I20240503 14:05:51.118853 18034 raft_consensus.cc:374] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:51.119091 18034 raft_consensus.cc:729] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db7b8c1d60b47b8b8737489cbd0ac93, State: Initialized, Role: FOLLOWER
I20240503 14:05:51.119184 18036 tablet_bootstrap.cc:654] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:51.119777 18034 consensus_queue.cc:260] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } }
I20240503 14:05:51.120190 18035 raft_consensus.cc:348] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } }
I20240503 14:05:51.121009 18035 raft_consensus.cc:374] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:51.121300 18035 raft_consensus.cc:729] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eceaf08d0aef4b71b49636ba75543e58, State: Initialized, Role: FOLLOWER
I20240503 14:05:51.122047 18035 consensus_queue.cc:260] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } }
I20240503 14:05:51.125283 18036 tablet_bootstrap.cc:492] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da: No bootstrap required, opened a new log
I20240503 14:05:51.125934 18036 ts_tablet_manager.cc:1367] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20240503 14:05:51.128223 18035 ts_tablet_manager.cc:1398] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240503 14:05:51.128835 18034 ts_tablet_manager.cc:1398] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20240503 14:05:51.130187 18036 raft_consensus.cc:348] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } }
I20240503 14:05:51.131153 18036 raft_consensus.cc:374] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:51.131484 18036 raft_consensus.cc:729] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25852080845240a8b99d8ba16719b0da, State: Initialized, Role: FOLLOWER
I20240503 14:05:51.132191 18036 consensus_queue.cc:260] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } }
I20240503 14:05:51.135207 18036 ts_tablet_manager.cc:1398] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s
W20240503 14:05:51.136081 17941 tablet.cc:2369] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:05:51.208397 18039 raft_consensus.cc:482] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:51.209167 18039 raft_consensus.cc:504] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } }
I20240503 14:05:51.211553 18039 leader_election.cc:288] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eceaf08d0aef4b71b49636ba75543e58 (127.13.158.67:39217), 25852080845240a8b99d8ba16719b0da (127.13.158.66:44789)
I20240503 14:05:51.222834 17989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd354b08a3fa47b0a039244adad9465e" candidate_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eceaf08d0aef4b71b49636ba75543e58" is_pre_election: true
I20240503 14:05:51.223395 17915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd354b08a3fa47b0a039244adad9465e" candidate_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25852080845240a8b99d8ba16719b0da" is_pre_election: true
I20240503 14:05:51.223664 17989 raft_consensus.cc:2454] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db7b8c1d60b47b8b8737489cbd0ac93 in term 0.
I20240503 14:05:51.224046 17915 raft_consensus.cc:2454] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db7b8c1d60b47b8b8737489cbd0ac93 in term 0.
I20240503 14:05:51.224951 17806 leader_election.cc:302] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8db7b8c1d60b47b8b8737489cbd0ac93, eceaf08d0aef4b71b49636ba75543e58; no voters:
I20240503 14:05:51.225783 18039 raft_consensus.cc:2781] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:51.226095 18039 raft_consensus.cc:482] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:51.226363 18039 raft_consensus.cc:3037] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:51.231873 18039 raft_consensus.cc:504] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } }
I20240503 14:05:51.233424 18039 leader_election.cc:288] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [CANDIDATE]: Term 1 election: Requested vote from peers eceaf08d0aef4b71b49636ba75543e58 (127.13.158.67:39217), 25852080845240a8b99d8ba16719b0da (127.13.158.66:44789)
I20240503 14:05:51.234529 17989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd354b08a3fa47b0a039244adad9465e" candidate_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eceaf08d0aef4b71b49636ba75543e58"
I20240503 14:05:51.234952 17915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd354b08a3fa47b0a039244adad9465e" candidate_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25852080845240a8b99d8ba16719b0da"
I20240503 14:05:51.235421 17989 raft_consensus.cc:3037] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:51.235495 17915 raft_consensus.cc:3037] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:51.242070 17915 raft_consensus.cc:2454] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db7b8c1d60b47b8b8737489cbd0ac93 in term 1.
I20240503 14:05:51.242523 17989 raft_consensus.cc:2454] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db7b8c1d60b47b8b8737489cbd0ac93 in term 1.
I20240503 14:05:51.243106 17806 leader_election.cc:302] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25852080845240a8b99d8ba16719b0da, 8db7b8c1d60b47b8b8737489cbd0ac93; no voters:
I20240503 14:05:51.243870 18039 raft_consensus.cc:2781] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:51.244971 18039 raft_consensus.cc:686] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 1 LEADER]: Becoming Leader. State: Replica: 8db7b8c1d60b47b8b8737489cbd0ac93, State: Running, Role: LEADER
I20240503 14:05:51.245651 18039 consensus_queue.cc:237] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } }
I20240503 14:05:51.252707 17737 catalog_manager.cc:5484] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8db7b8c1d60b47b8b8737489cbd0ac93 (127.13.158.65). New cstate: current_term: 1 leader_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:51.279899 13945 tablet_server.cc:178] TabletServer@127.13.158.65:0 shutting down...
I20240503 14:05:51.297513 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:05:51.298333 13945 tablet_replica.cc:330] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93: stopping tablet replica
I20240503 14:05:51.299196 13945 raft_consensus.cc:2229] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:05:51.299914 13945 pending_rounds.cc:62] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93: Trying to abort 1 pending ops.
I20240503 14:05:51.300187 13945 pending_rounds.cc:68] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023596139509141504 op_type: NO_OP noop_request { }
I20240503 14:05:51.300508 13945 raft_consensus.cc:2866] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:05:51.300773 13945 raft_consensus.cc:2258] T bd354b08a3fa47b0a039244adad9465e P 8db7b8c1d60b47b8b8737489cbd0ac93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:51.320678 13945 tablet_server.cc:194] TabletServer@127.13.158.65:0 shutdown complete.
I20240503 14:05:51.614670 17791 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:51.616147 17791 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:51.616461 17791 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:05:53.194885 18047 raft_consensus.cc:482] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:53.195470 18047 raft_consensus.cc:504] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } }
I20240503 14:05:53.197824 18047 leader_election.cc:288] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8db7b8c1d60b47b8b8737489cbd0ac93 (127.13.158.65:38369), eceaf08d0aef4b71b49636ba75543e58 (127.13.158.67:39217)
W20240503 14:05:53.203558 17880 leader_election.cc:334] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8db7b8c1d60b47b8b8737489cbd0ac93 (127.13.158.65:38369): Network error: Client connection negotiation failed: client connection to 127.13.158.65:38369: connect: Connection refused (error 111)
I20240503 14:05:53.209131 17989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd354b08a3fa47b0a039244adad9465e" candidate_uuid: "25852080845240a8b99d8ba16719b0da" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eceaf08d0aef4b71b49636ba75543e58" is_pre_election: true
I20240503 14:05:53.210058 17989 raft_consensus.cc:2454] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25852080845240a8b99d8ba16719b0da in term 1.
I20240503 14:05:53.211385 17881 leader_election.cc:302] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25852080845240a8b99d8ba16719b0da, eceaf08d0aef4b71b49636ba75543e58; no voters: 8db7b8c1d60b47b8b8737489cbd0ac93
I20240503 14:05:53.212314 18047 raft_consensus.cc:2781] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:05:53.212694 18047 raft_consensus.cc:482] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:53.213011 18047 raft_consensus.cc:3037] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:05:53.219518 18047 raft_consensus.cc:504] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } }
I20240503 14:05:53.221233 18047 leader_election.cc:288] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [CANDIDATE]: Term 2 election: Requested vote from peers 8db7b8c1d60b47b8b8737489cbd0ac93 (127.13.158.65:38369), eceaf08d0aef4b71b49636ba75543e58 (127.13.158.67:39217)
I20240503 14:05:53.222513 17989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd354b08a3fa47b0a039244adad9465e" candidate_uuid: "25852080845240a8b99d8ba16719b0da" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eceaf08d0aef4b71b49636ba75543e58"
I20240503 14:05:53.223121 17989 raft_consensus.cc:3037] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:05:53.224862 17880 leader_election.cc:334] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8db7b8c1d60b47b8b8737489cbd0ac93 (127.13.158.65:38369): Network error: Client connection negotiation failed: client connection to 127.13.158.65:38369: connect: Connection refused (error 111)
I20240503 14:05:53.228924 17989 raft_consensus.cc:2454] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25852080845240a8b99d8ba16719b0da in term 2.
I20240503 14:05:53.230141 17881 leader_election.cc:302] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25852080845240a8b99d8ba16719b0da, eceaf08d0aef4b71b49636ba75543e58; no voters: 8db7b8c1d60b47b8b8737489cbd0ac93
I20240503 14:05:53.230904 18047 raft_consensus.cc:2781] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:05:53.232120 18047 raft_consensus.cc:686] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 2 LEADER]: Becoming Leader. State: Replica: 25852080845240a8b99d8ba16719b0da, State: Running, Role: LEADER
I20240503 14:05:53.232787 18047 consensus_queue.cc:237] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } }
I20240503 14:05:53.240451 17737 catalog_manager.cc:5484] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da reported cstate change: term changed from 1 to 2, leader changed from 8db7b8c1d60b47b8b8737489cbd0ac93 (127.13.158.65) to 25852080845240a8b99d8ba16719b0da (127.13.158.66). New cstate: current_term: 2 leader_uuid: "25852080845240a8b99d8ba16719b0da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db7b8c1d60b47b8b8737489cbd0ac93" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 38369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25852080845240a8b99d8ba16719b0da" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 44789 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:53.617395 17791 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:53.618862 17791 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:53.619208 17791 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 14:05:53.635726 17880 consensus_peers.cc:485] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da -> Peer 8db7b8c1d60b47b8b8737489cbd0ac93 (127.13.158.65:38369): Couldn't send request to peer 8db7b8c1d60b47b8b8737489cbd0ac93. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:38369: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:53.708237 17989 raft_consensus.cc:1261] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [term 2 FOLLOWER]: Refusing update from remote peer 25852080845240a8b99d8ba16719b0da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 14:05:53.709671 18047 consensus_queue.cc:1035] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [LEADER]: Connected to new peer: Peer: permanent_uuid: "eceaf08d0aef4b71b49636ba75543e58" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 39217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:05:54.244830 17880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.158.65:38369: connect: Connection refused (error 111) [suppressed 3 similar messages]
I20240503 14:05:55.620304 17791 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:55.621523 17791 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:55.621793 17791 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 14:05:56.108686 17880 consensus_peers.cc:485] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da -> Peer 8db7b8c1d60b47b8b8737489cbd0ac93 (127.13.158.65:38369): Couldn't send request to peer 8db7b8c1d60b47b8b8737489cbd0ac93. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:38369: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:05:57.623088 17791 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:57.624739 17791 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:57.625097 17791 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 14:05:58.718077 17880 consensus_peers.cc:485] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da -> Peer 8db7b8c1d60b47b8b8737489cbd0ac93 (127.13.158.65:38369): Couldn't send request to peer 8db7b8c1d60b47b8b8737489cbd0ac93. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:38369: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:59.508634 17769 debug-util.cc:398] Leaking SignalData structure 0x7b0800215380 after lost signal to thread 13948
W20240503 14:05:59.509555 17769 debug-util.cc:398] Leaking SignalData structure 0x7b08000db940 after lost signal to thread 17772
W20240503 14:05:59.510541 17769 debug-util.cc:398] Leaking SignalData structure 0x7b08000cca60 after lost signal to thread 17939
W20240503 14:05:59.511395 17769 debug-util.cc:398] Leaking SignalData structure 0x7b08001f9fa0 after lost signal to thread 18013
W20240503 14:05:59.529453 17880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.158.65:38369: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 14:05:59.625998 17791 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:05:59.627281 17791 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:05:59.627573 17791 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 14:06:00.908145 17880 consensus_peers.cc:485] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da -> Peer 8db7b8c1d60b47b8b8737489cbd0ac93 (127.13.158.65:38369): Couldn't send request to peer 8db7b8c1d60b47b8b8737489cbd0ac93. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:38369: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:06:01.628456 17791 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:06:01.629781 17791 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:06:01.630071 17791 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:06:02.416651 13945 tablet_server.cc:178] TabletServer@127.13.158.66:0 shutting down...
I20240503 14:06:02.438930 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:06:02.439618 13945 tablet_replica.cc:330] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da: stopping tablet replica
I20240503 14:06:02.440237 13945 raft_consensus.cc:2229] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 2 LEADER]: Raft consensus shutting down.
I20240503 14:06:02.441291 13945 raft_consensus.cc:2258] T bd354b08a3fa47b0a039244adad9465e P 25852080845240a8b99d8ba16719b0da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:02.461721 13945 tablet_server.cc:194] TabletServer@127.13.158.66:0 shutdown complete.
I20240503 14:06:02.474589 13945 tablet_server.cc:178] TabletServer@127.13.158.67:0 shutting down...
I20240503 14:06:02.495358 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:06:02.496031 13945 tablet_replica.cc:330] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58: stopping tablet replica
I20240503 14:06:02.496572 13945 raft_consensus.cc:2229] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:02.497068 13945 raft_consensus.cc:2258] T bd354b08a3fa47b0a039244adad9465e P eceaf08d0aef4b71b49636ba75543e58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:02.516736 13945 tablet_server.cc:194] TabletServer@127.13.158.67:0 shutdown complete.
I20240503 14:06:02.527293 13945 master.cc:537] Master@127.13.158.126:39803 shutting down...
I20240503 14:06:02.544974 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:06:02.545643 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:02.545970 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P ee38cd4a3e2b4f6999795de896d76f3e: stopping tablet replica
I20240503 14:06:02.580772 13945 master.cc:552] Master@127.13.158.126:39803 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (13156 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20240503 14:06:02.618535 13945 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.13.158.126:40425
I20240503 14:06:02.619642 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:06:02.626567 18082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:02.627451 18083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:02.628445 18085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:02.628988 13945 server_base.cc:1034] running on GCE node
I20240503 14:06:02.629854 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:02.630055 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:02.630218 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745162630200 us; error 0 us; skew 500 ppm
I20240503 14:06:02.630776 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:02.633314 13945 webserver.cc:460] Webserver started at http://127.13.158.126:41791/ using document root <none> and password file <none>
I20240503 14:06:02.633800 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:02.633961 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:06:02.634256 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:06:02.635717 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/master-0-root/instance:
uuid: "34d9f7b5283843c7a77c3293ffe384a6"
format_stamp: "Formatted at 2024-05-03 14:06:02 on dist-test-slave-jm3r"
I20240503 14:06:02.640601 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:06:02.644721 18090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:02.645830 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:06:02.646324 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/master-0-root
uuid: "34d9f7b5283843c7a77c3293ffe384a6"
format_stamp: "Formatted at 2024-05-03 14:06:02 on dist-test-slave-jm3r"
I20240503 14:06:02.646804 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:02.672304 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:02.673509 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:02.711205 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.126:40425
I20240503 14:06:02.711328 18141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.126:40425 every 8 connection(s)
I20240503 14:06:02.715975 18142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:02.728788 18142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6: Bootstrap starting.
I20240503 14:06:02.733207 18142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:02.737649 18142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6: No bootstrap required, opened a new log
I20240503 14:06:02.739760 18142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d9f7b5283843c7a77c3293ffe384a6" member_type: VOTER }
I20240503 14:06:02.740252 18142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:02.740490 18142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d9f7b5283843c7a77c3293ffe384a6, State: Initialized, Role: FOLLOWER
I20240503 14:06:02.741088 18142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d9f7b5283843c7a77c3293ffe384a6" member_type: VOTER }
I20240503 14:06:02.741576 18142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:06:02.741793 18142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:06:02.742046 18142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:02.746821 18142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d9f7b5283843c7a77c3293ffe384a6" member_type: VOTER }
I20240503 14:06:02.747416 18142 leader_election.cc:302] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34d9f7b5283843c7a77c3293ffe384a6; no voters:
I20240503 14:06:02.748831 18142 leader_election.cc:288] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:06:02.749444 18145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:06:02.751016 18145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [term 1 LEADER]: Becoming Leader. State: Replica: 34d9f7b5283843c7a77c3293ffe384a6, State: Running, Role: LEADER
I20240503 14:06:02.751868 18145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d9f7b5283843c7a77c3293ffe384a6" member_type: VOTER }
I20240503 14:06:02.752087 18142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:06:02.757442 18146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34d9f7b5283843c7a77c3293ffe384a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d9f7b5283843c7a77c3293ffe384a6" member_type: VOTER } }
I20240503 14:06:02.758459 18146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [sys.catalog]: This master's current role is: LEADER
I20240503 14:06:02.757567 18147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34d9f7b5283843c7a77c3293ffe384a6. Latest consensus state: current_term: 1 leader_uuid: "34d9f7b5283843c7a77c3293ffe384a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d9f7b5283843c7a77c3293ffe384a6" member_type: VOTER } }
I20240503 14:06:02.759449 18147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [sys.catalog]: This master's current role is: LEADER
I20240503 14:06:02.761315 18154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:06:02.768719 18154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:06:02.769740 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:06:02.777799 18154 catalog_manager.cc:1347] Generated new cluster ID: d69013e2e38f48f38ef5b07189f51a35
I20240503 14:06:02.778059 18154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:06:02.790022 18154 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:06:02.791430 18154 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:06:02.816206 18154 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6: Generated new TSK 0
I20240503 14:06:02.816797 18154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:06:02.837013 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:06:02.844417 18163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:02.845408 18164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:02.846549 13945 server_base.cc:1034] running on GCE node
W20240503 14:06:02.847299 18166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:02.848138 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:02.848366 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:02.848520 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745162848502 us; error 0 us; skew 500 ppm
I20240503 14:06:02.849061 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:02.851629 13945 webserver.cc:460] Webserver started at http://127.13.158.65:35041/ using document root <none> and password file <none>
I20240503 14:06:02.852129 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:02.852315 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:06:02.852579 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:06:02.853889 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-0-root/instance:
uuid: "5c138f5b79ac428886fcaeb9ff725dd9"
format_stamp: "Formatted at 2024-05-03 14:06:02 on dist-test-slave-jm3r"
I20240503 14:06:02.858508 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:06:02.862008 18171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:02.862864 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:06:02.863173 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-0-root
uuid: "5c138f5b79ac428886fcaeb9ff725dd9"
format_stamp: "Formatted at 2024-05-03 14:06:02 on dist-test-slave-jm3r"
I20240503 14:06:02.863456 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:02.883832 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:02.884997 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:02.886482 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:06:02.889005 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:06:02.889240 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:02.889477 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:06:02.889642 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:02.933266 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.65:34717
I20240503 14:06:02.933372 18233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.65:34717 every 8 connection(s)
I20240503 14:06:02.938056 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:06:02.948793 18239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:02.948992 18238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:02.952224 18234 heartbeater.cc:347] Connected to a master server at 127.13.158.126:40425
I20240503 14:06:02.952811 18234 heartbeater.cc:464] Registering TS with master...
I20240503 14:06:02.952972 13945 server_base.cc:1034] running on GCE node
I20240503 14:06:02.953824 18234 heartbeater.cc:511] Master 127.13.158.126:40425 requested a full tablet report, sending...
W20240503 14:06:02.954044 18241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:02.955195 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:02.955436 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:02.955658 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745162955635 us; error 0 us; skew 500 ppm
I20240503 14:06:02.956568 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:02.956451 18107 ts_manager.cc:194] Registered new tserver with Master: 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717)
I20240503 14:06:02.959165 18107 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47812
I20240503 14:06:02.964118 13945 webserver.cc:460] Webserver started at http://127.13.158.66:41667/ using document root <none> and password file <none>
I20240503 14:06:02.964884 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:02.965126 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:06:02.965415 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:06:02.967172 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-1-root/instance:
uuid: "fede8e81a5df42febf80ef0911d772d1"
format_stamp: "Formatted at 2024-05-03 14:06:02 on dist-test-slave-jm3r"
I20240503 14:06:02.975111 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20240503 14:06:02.978806 18246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:02.979648 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:06:02.979930 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-1-root
uuid: "fede8e81a5df42febf80ef0911d772d1"
format_stamp: "Formatted at 2024-05-03 14:06:02 on dist-test-slave-jm3r"
I20240503 14:06:02.980226 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:02.988956 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:02.990176 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:02.991647 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:06:02.993960 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:06:02.994170 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:02.994393 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:06:02.994609 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 14:06:03.039376 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.66:41189
I20240503 14:06:03.039474 18308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.66:41189 every 8 connection(s)
I20240503 14:06:03.044080 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:06:03.052099 18312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:03.053103 18313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:03.054769 18315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:03.057037 13945 server_base.cc:1034] running on GCE node
I20240503 14:06:03.057303 18309 heartbeater.cc:347] Connected to a master server at 127.13.158.126:40425
I20240503 14:06:03.057745 18309 heartbeater.cc:464] Registering TS with master...
I20240503 14:06:03.058105 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:03.058378 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:03.058627 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745163058604 us; error 0 us; skew 500 ppm
I20240503 14:06:03.058789 18309 heartbeater.cc:511] Master 127.13.158.126:40425 requested a full tablet report, sending...
I20240503 14:06:03.059487 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:03.061170 18107 ts_manager.cc:194] Registered new tserver with Master: fede8e81a5df42febf80ef0911d772d1 (127.13.158.66:41189)
I20240503 14:06:03.062527 13945 webserver.cc:460] Webserver started at http://127.13.158.67:36497/ using document root <none> and password file <none>
I20240503 14:06:03.062814 18107 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47824
I20240503 14:06:03.063045 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:03.063308 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:06:03.063553 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:06:03.065198 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-2-root/instance:
uuid: "43858e199ddd414fb44cebc2bf56b431"
format_stamp: "Formatted at 2024-05-03 14:06:03 on dist-test-slave-jm3r"
I20240503 14:06:03.070062 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240503 14:06:03.073716 18320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:03.074595 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:06:03.074858 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-2-root
uuid: "43858e199ddd414fb44cebc2bf56b431"
format_stamp: "Formatted at 2024-05-03 14:06:03 on dist-test-slave-jm3r"
I20240503 14:06:03.075099 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:03.103545 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:03.104650 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:03.106050 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:06:03.108400 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:06:03.108593 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:03.108847 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:06:03.109004 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:03.149987 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.67:34431
I20240503 14:06:03.150116 18382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.67:34431 every 8 connection(s)
I20240503 14:06:03.163156 18383 heartbeater.cc:347] Connected to a master server at 127.13.158.126:40425
I20240503 14:06:03.163609 18383 heartbeater.cc:464] Registering TS with master...
I20240503 14:06:03.164541 18383 heartbeater.cc:511] Master 127.13.158.126:40425 requested a full tablet report, sending...
I20240503 14:06:03.166390 18107 ts_manager.cc:194] Registered new tserver with Master: 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67:34431)
I20240503 14:06:03.166890 13945 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013805162s
I20240503 14:06:03.168145 18107 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47830
I20240503 14:06:03.963030 18234 heartbeater.cc:503] Master 127.13.158.126:40425 was elected leader, sending a full tablet report...
I20240503 14:06:04.065685 18309 heartbeater.cc:503] Master 127.13.158.126:40425 was elected leader, sending a full tablet report...
I20240503 14:06:04.170990 18383 heartbeater.cc:503] Master 127.13.158.126:40425 was elected leader, sending a full tablet report...
I20240503 14:06:04.200327 13945 test_util.cc:268] Using random seed: 946108759
I20240503 14:06:04.224328 18107 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47832:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:06:04.226735 18107 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:06:04.284261 18199 tablet_service.cc:1467] Processing CreateTablet for tablet 4e478e48cb48476e9cde5fd215be0e02 (DEFAULT_TABLE table=test-workload [id=2beb719d24b14d4cb9e866bcac57bede]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 14:06:04.285754 18199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e478e48cb48476e9cde5fd215be0e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:04.287708 18196 tablet_service.cc:1467] Processing CreateTablet for tablet 2f053f9a1b9e41f68fc778572aaf64cf (DEFAULT_TABLE table=test-workload [id=2beb719d24b14d4cb9e866bcac57bede]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 14:06:04.289069 18196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f053f9a1b9e41f68fc778572aaf64cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:04.291563 18198 tablet_service.cc:1467] Processing CreateTablet for tablet 2540f6226da9468195f0ded09c888a6b (DEFAULT_TABLE table=test-workload [id=2beb719d24b14d4cb9e866bcac57bede]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 14:06:04.292836 18198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2540f6226da9468195f0ded09c888a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:04.293056 18274 tablet_service.cc:1467] Processing CreateTablet for tablet 4e478e48cb48476e9cde5fd215be0e02 (DEFAULT_TABLE table=test-workload [id=2beb719d24b14d4cb9e866bcac57bede]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 14:06:04.294348 18274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e478e48cb48476e9cde5fd215be0e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:04.294790 18272 tablet_service.cc:1467] Processing CreateTablet for tablet c372eadbe3de4dda8c100977d1f017a7 (DEFAULT_TABLE table=test-workload [id=2beb719d24b14d4cb9e866bcac57bede]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 14:06:04.295888 18272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c372eadbe3de4dda8c100977d1f017a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:04.295598 18197 tablet_service.cc:1467] Processing CreateTablet for tablet c372eadbe3de4dda8c100977d1f017a7 (DEFAULT_TABLE table=test-workload [id=2beb719d24b14d4cb9e866bcac57bede]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 14:06:04.296952 18197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c372eadbe3de4dda8c100977d1f017a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:04.299396 18271 tablet_service.cc:1467] Processing CreateTablet for tablet 2f053f9a1b9e41f68fc778572aaf64cf (DEFAULT_TABLE table=test-workload [id=2beb719d24b14d4cb9e866bcac57bede]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 14:06:04.300588 18271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f053f9a1b9e41f68fc778572aaf64cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:04.293521 18273 tablet_service.cc:1467] Processing CreateTablet for tablet 2540f6226da9468195f0ded09c888a6b (DEFAULT_TABLE table=test-workload [id=2beb719d24b14d4cb9e866bcac57bede]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 14:06:04.303345 18273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2540f6226da9468195f0ded09c888a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:04.320391 18403 tablet_bootstrap.cc:492] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1: Bootstrap starting.
I20240503 14:06:04.333686 18347 tablet_service.cc:1467] Processing CreateTablet for tablet 2540f6226da9468195f0ded09c888a6b (DEFAULT_TABLE table=test-workload [id=2beb719d24b14d4cb9e866bcac57bede]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 14:06:04.334268 18348 tablet_service.cc:1467] Processing CreateTablet for tablet 4e478e48cb48476e9cde5fd215be0e02 (DEFAULT_TABLE table=test-workload [id=2beb719d24b14d4cb9e866bcac57bede]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 14:06:04.335695 18348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e478e48cb48476e9cde5fd215be0e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:04.336864 18345 tablet_service.cc:1467] Processing CreateTablet for tablet 2f053f9a1b9e41f68fc778572aaf64cf (DEFAULT_TABLE table=test-workload [id=2beb719d24b14d4cb9e866bcac57bede]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 14:06:04.338135 18345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f053f9a1b9e41f68fc778572aaf64cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:04.335702 18346 tablet_service.cc:1467] Processing CreateTablet for tablet c372eadbe3de4dda8c100977d1f017a7 (DEFAULT_TABLE table=test-workload [id=2beb719d24b14d4cb9e866bcac57bede]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 14:06:04.343972 18346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c372eadbe3de4dda8c100977d1f017a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:04.345876 18347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2540f6226da9468195f0ded09c888a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:04.346905 18403 tablet_bootstrap.cc:654] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:04.347328 18404 tablet_bootstrap.cc:492] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9: Bootstrap starting.
I20240503 14:06:04.362599 18403 tablet_bootstrap.cc:492] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1: No bootstrap required, opened a new log
I20240503 14:06:04.363204 18403 ts_tablet_manager.cc:1367] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.007s
I20240503 14:06:04.366057 18403 raft_consensus.cc:348] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.366703 18404 tablet_bootstrap.cc:654] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:04.366878 18403 raft_consensus.cc:374] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:04.367215 18403 raft_consensus.cc:729] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fede8e81a5df42febf80ef0911d772d1, State: Initialized, Role: FOLLOWER
I20240503 14:06:04.367861 18403 consensus_queue.cc:260] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.370473 18403 ts_tablet_manager.cc:1398] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:06:04.371569 18403 tablet_bootstrap.cc:492] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1: Bootstrap starting.
I20240503 14:06:04.373699 18406 tablet_bootstrap.cc:492] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431: Bootstrap starting.
I20240503 14:06:04.377889 18403 tablet_bootstrap.cc:654] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:04.382598 18406 tablet_bootstrap.cc:654] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:04.384178 18404 tablet_bootstrap.cc:492] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9: No bootstrap required, opened a new log
I20240503 14:06:04.384748 18404 ts_tablet_manager.cc:1367] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.004s
I20240503 14:06:04.387574 18404 raft_consensus.cc:348] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.388401 18404 raft_consensus.cc:374] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:04.388689 18404 raft_consensus.cc:729] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c138f5b79ac428886fcaeb9ff725dd9, State: Initialized, Role: FOLLOWER
I20240503 14:06:04.389511 18404 consensus_queue.cc:260] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.392189 18403 tablet_bootstrap.cc:492] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1: No bootstrap required, opened a new log
I20240503 14:06:04.392704 18404 ts_tablet_manager.cc:1398] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240503 14:06:04.392745 18406 tablet_bootstrap.cc:492] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431: No bootstrap required, opened a new log
I20240503 14:06:04.392715 18403 ts_tablet_manager.cc:1367] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.004s
I20240503 14:06:04.393529 18406 ts_tablet_manager.cc:1367] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
I20240503 14:06:04.393982 18404 tablet_bootstrap.cc:492] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9: Bootstrap starting.
I20240503 14:06:04.395998 18403 raft_consensus.cc:348] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.396363 18406 raft_consensus.cc:348] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.396803 18403 raft_consensus.cc:374] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:04.397190 18406 raft_consensus.cc:374] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:04.397236 18403 raft_consensus.cc:729] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fede8e81a5df42febf80ef0911d772d1, State: Initialized, Role: FOLLOWER
I20240503 14:06:04.397732 18406 raft_consensus.cc:729] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43858e199ddd414fb44cebc2bf56b431, State: Initialized, Role: FOLLOWER
I20240503 14:06:04.398380 18403 consensus_queue.cc:260] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.398537 18406 consensus_queue.cc:260] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.400465 18404 tablet_bootstrap.cc:654] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:04.408190 18406 ts_tablet_manager.cc:1398] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20240503 14:06:04.408758 18403 ts_tablet_manager.cc:1398] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1: Time spent starting tablet: real 0.015s user 0.003s sys 0.003s
I20240503 14:06:04.409305 18406 tablet_bootstrap.cc:492] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431: Bootstrap starting.
I20240503 14:06:04.411041 18404 tablet_bootstrap.cc:492] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9: No bootstrap required, opened a new log
I20240503 14:06:04.411550 18404 ts_tablet_manager.cc:1367] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.000s
I20240503 14:06:04.413755 18403 tablet_bootstrap.cc:492] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1: Bootstrap starting.
I20240503 14:06:04.414211 18404 raft_consensus.cc:348] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.414887 18407 raft_consensus.cc:482] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:04.415179 18404 raft_consensus.cc:374] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:04.415407 18406 tablet_bootstrap.cc:654] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:04.415580 18404 raft_consensus.cc:729] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c138f5b79ac428886fcaeb9ff725dd9, State: Initialized, Role: FOLLOWER
I20240503 14:06:04.416433 18404 consensus_queue.cc:260] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.420548 18403 tablet_bootstrap.cc:654] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:04.421036 18404 ts_tablet_manager.cc:1398] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:06:04.422183 18404 tablet_bootstrap.cc:492] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9: Bootstrap starting.
I20240503 14:06:04.427232 18406 tablet_bootstrap.cc:492] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431: No bootstrap required, opened a new log
I20240503 14:06:04.427789 18406 ts_tablet_manager.cc:1367] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.005s
I20240503 14:06:04.428032 18404 tablet_bootstrap.cc:654] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:04.430594 18406 raft_consensus.cc:348] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.431442 18406 raft_consensus.cc:374] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:04.431742 18406 raft_consensus.cc:729] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43858e199ddd414fb44cebc2bf56b431, State: Initialized, Role: FOLLOWER
I20240503 14:06:04.432751 18406 consensus_queue.cc:260] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.434986 18407 raft_consensus.cc:504] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.437093 18406 ts_tablet_manager.cc:1398] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20240503 14:06:04.438262 18406 tablet_bootstrap.cc:492] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431: Bootstrap starting.
I20240503 14:06:04.441685 18407 leader_election.cc:288] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717), 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67:34431)
I20240503 14:06:04.450599 18403 tablet_bootstrap.cc:492] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1: No bootstrap required, opened a new log
I20240503 14:06:04.451385 18403 ts_tablet_manager.cc:1367] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.016s
I20240503 14:06:04.454456 18403 raft_consensus.cc:348] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.455371 18403 raft_consensus.cc:374] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:04.455732 18403 raft_consensus.cc:729] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fede8e81a5df42febf80ef0911d772d1, State: Initialized, Role: FOLLOWER
I20240503 14:06:04.456562 18403 consensus_queue.cc:260] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.457250 18404 tablet_bootstrap.cc:492] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9: No bootstrap required, opened a new log
I20240503 14:06:04.457845 18404 ts_tablet_manager.cc:1367] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9: Time spent bootstrapping tablet: real 0.036s user 0.020s sys 0.012s
I20240503 14:06:04.459159 18403 ts_tablet_manager.cc:1398] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240503 14:06:04.460404 18403 tablet_bootstrap.cc:492] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1: Bootstrap starting.
I20240503 14:06:04.460749 18404 raft_consensus.cc:348] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.461588 18404 raft_consensus.cc:374] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:04.461912 18404 raft_consensus.cc:729] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c138f5b79ac428886fcaeb9ff725dd9, State: Initialized, Role: FOLLOWER
I20240503 14:06:04.463068 18406 tablet_bootstrap.cc:654] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:04.462690 18404 consensus_queue.cc:260] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.465677 18404 ts_tablet_manager.cc:1398] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20240503 14:06:04.467042 18404 tablet_bootstrap.cc:492] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9: Bootstrap starting.
I20240503 14:06:04.467435 18403 tablet_bootstrap.cc:654] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:04.471617 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053f9a1b9e41f68fc778572aaf64cf" candidate_uuid: "fede8e81a5df42febf80ef0911d772d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43858e199ddd414fb44cebc2bf56b431" is_pre_election: true
I20240503 14:06:04.472762 18404 tablet_bootstrap.cc:654] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9: Neither blocks nor log segments found. Creating new log.
W20240503 14:06:04.473527 18249 leader_election.cc:341] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67:34431): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:06:04.484472 18407 raft_consensus.cc:482] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:04.484872 18404 tablet_bootstrap.cc:492] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9: No bootstrap required, opened a new log
I20240503 14:06:04.485422 18404 ts_tablet_manager.cc:1367] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20240503 14:06:04.485029 18407 raft_consensus.cc:504] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.486209 18412 raft_consensus.cc:482] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:04.486712 18412 raft_consensus.cc:504] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.488246 18404 raft_consensus.cc:348] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.489032 18404 raft_consensus.cc:374] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:04.488993 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e478e48cb48476e9cde5fd215be0e02" candidate_uuid: "fede8e81a5df42febf80ef0911d772d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43858e199ddd414fb44cebc2bf56b431" is_pre_election: true
I20240503 14:06:04.489552 18404 raft_consensus.cc:729] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c138f5b79ac428886fcaeb9ff725dd9, State: Initialized, Role: FOLLOWER
I20240503 14:06:04.489869 18358 raft_consensus.cc:2454] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fede8e81a5df42febf80ef0911d772d1 in term 0.
I20240503 14:06:04.490537 18404 consensus_queue.cc:260] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.489310 18412 leader_election.cc:288] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717), fede8e81a5df42febf80ef0911d772d1 (127.13.158.66:41189)
I20240503 14:06:04.491360 18249 leader_election.cc:302] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43858e199ddd414fb44cebc2bf56b431, fede8e81a5df42febf80ef0911d772d1; no voters:
I20240503 14:06:04.493513 18423 raft_consensus.cc:2781] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:06:04.494392 18423 raft_consensus.cc:482] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:04.495249 18423 raft_consensus.cc:3037] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:04.496215 18404 ts_tablet_manager.cc:1398] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240503 14:06:04.498917 18407 leader_election.cc:288] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717), 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67:34431)
I20240503 14:06:04.498960 18406 tablet_bootstrap.cc:492] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431: No bootstrap required, opened a new log
I20240503 14:06:04.511766 18406 ts_tablet_manager.cc:1367] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.001s
I20240503 14:06:04.512818 18423 raft_consensus.cc:504] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.517557 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e478e48cb48476e9cde5fd215be0e02" candidate_uuid: "fede8e81a5df42febf80ef0911d772d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43858e199ddd414fb44cebc2bf56b431"
I20240503 14:06:04.518394 18358 raft_consensus.cc:3037] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:04.518891 18406 raft_consensus.cc:348] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.519640 18406 raft_consensus.cc:374] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:04.519930 18406 raft_consensus.cc:729] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43858e199ddd414fb44cebc2bf56b431, State: Initialized, Role: FOLLOWER
I20240503 14:06:04.520642 18406 consensus_queue.cc:260] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.522992 18406 ts_tablet_manager.cc:1398] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240503 14:06:04.523720 18207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e478e48cb48476e9cde5fd215be0e02" candidate_uuid: "fede8e81a5df42febf80ef0911d772d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c138f5b79ac428886fcaeb9ff725dd9"
I20240503 14:06:04.524123 18406 tablet_bootstrap.cc:492] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431: Bootstrap starting.
I20240503 14:06:04.524459 18207 raft_consensus.cc:3037] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:04.517679 18209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053f9a1b9e41f68fc778572aaf64cf" candidate_uuid: "fede8e81a5df42febf80ef0911d772d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" is_pre_election: true
I20240503 14:06:04.526598 18411 raft_consensus.cc:482] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:04.523065 18208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e478e48cb48476e9cde5fd215be0e02" candidate_uuid: "fede8e81a5df42febf80ef0911d772d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" is_pre_election: true
I20240503 14:06:04.527199 18411 raft_consensus.cc:504] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.527674 18209 raft_consensus.cc:2454] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fede8e81a5df42febf80ef0911d772d1 in term 0.
I20240503 14:06:04.529084 18247 leader_election.cc:302] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c138f5b79ac428886fcaeb9ff725dd9, fede8e81a5df42febf80ef0911d772d1; no voters: 43858e199ddd414fb44cebc2bf56b431
I20240503 14:06:04.530062 18407 raft_consensus.cc:2781] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:06:04.530443 18407 raft_consensus.cc:482] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:04.530793 18407 raft_consensus.cc:3037] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:04.532495 18411 leader_election.cc:288] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fede8e81a5df42febf80ef0911d772d1 (127.13.158.66:41189), 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67:34431)
I20240503 14:06:04.537981 18207 raft_consensus.cc:2454] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fede8e81a5df42febf80ef0911d772d1 in term 1.
I20240503 14:06:04.538100 18208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c372eadbe3de4dda8c100977d1f017a7" candidate_uuid: "43858e199ddd414fb44cebc2bf56b431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" is_pre_election: true
I20240503 14:06:04.541846 18208 raft_consensus.cc:2454] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43858e199ddd414fb44cebc2bf56b431 in term 0.
I20240503 14:06:04.541949 18407 raft_consensus.cc:504] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.543370 18321 leader_election.cc:302] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43858e199ddd414fb44cebc2bf56b431, 5c138f5b79ac428886fcaeb9ff725dd9; no voters:
I20240503 14:06:04.541096 18358 raft_consensus.cc:2454] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fede8e81a5df42febf80ef0911d772d1 in term 1.
I20240503 14:06:04.544293 18412 raft_consensus.cc:2781] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:06:04.544749 18412 raft_consensus.cc:482] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:04.545087 18412 raft_consensus.cc:3037] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:04.545557 18407 leader_election.cc:288] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 1 election: Requested vote from peers 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717), 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67:34431)
I20240503 14:06:04.546881 18423 leader_election.cc:288] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 1 election: Requested vote from peers 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717), 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67:34431)
I20240503 14:06:04.548243 18406 tablet_bootstrap.cc:654] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:04.548832 18403 tablet_bootstrap.cc:492] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1: No bootstrap required, opened a new log
I20240503 14:06:04.549130 18249 leader_election.cc:302] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43858e199ddd414fb44cebc2bf56b431, fede8e81a5df42febf80ef0911d772d1; no voters:
I20240503 14:06:04.549364 18403 ts_tablet_manager.cc:1367] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1: Time spent bootstrapping tablet: real 0.089s user 0.021s sys 0.020s
I20240503 14:06:04.553411 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053f9a1b9e41f68fc778572aaf64cf" candidate_uuid: "fede8e81a5df42febf80ef0911d772d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43858e199ddd414fb44cebc2bf56b431"
W20240503 14:06:04.555126 18249 leader_election.cc:341] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67:34431): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:06:04.556782 18403 raft_consensus.cc:348] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.557669 18403 raft_consensus.cc:374] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:04.557987 18403 raft_consensus.cc:729] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fede8e81a5df42febf80ef0911d772d1, State: Initialized, Role: FOLLOWER
I20240503 14:06:04.560983 18423 raft_consensus.cc:2781] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:06:04.565467 18423 raft_consensus.cc:686] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 1 LEADER]: Becoming Leader. State: Replica: fede8e81a5df42febf80ef0911d772d1, State: Running, Role: LEADER
I20240503 14:06:04.565889 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c372eadbe3de4dda8c100977d1f017a7" candidate_uuid: "43858e199ddd414fb44cebc2bf56b431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fede8e81a5df42febf80ef0911d772d1" is_pre_election: true
I20240503 14:06:04.566915 18207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053f9a1b9e41f68fc778572aaf64cf" candidate_uuid: "fede8e81a5df42febf80ef0911d772d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c138f5b79ac428886fcaeb9ff725dd9"
I20240503 14:06:04.565119 18412 raft_consensus.cc:504] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.566509 18423 consensus_queue.cc:237] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.567675 18207 raft_consensus.cc:3037] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:04.569593 18412 leader_election.cc:288] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [CANDIDATE]: Term 1 election: Requested vote from peers 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717), fede8e81a5df42febf80ef0911d772d1 (127.13.158.66:41189)
I20240503 14:06:04.567709 18403 consensus_queue.cc:260] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.570804 18208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c372eadbe3de4dda8c100977d1f017a7" candidate_uuid: "43858e199ddd414fb44cebc2bf56b431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c138f5b79ac428886fcaeb9ff725dd9"
I20240503 14:06:04.571307 18284 raft_consensus.cc:2454] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43858e199ddd414fb44cebc2bf56b431 in term 0.
I20240503 14:06:04.571806 18208 raft_consensus.cc:3037] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:04.576388 18207 raft_consensus.cc:2454] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fede8e81a5df42febf80ef0911d772d1 in term 1.
I20240503 14:06:04.577487 18247 leader_election.cc:302] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c138f5b79ac428886fcaeb9ff725dd9, fede8e81a5df42febf80ef0911d772d1; no voters: 43858e199ddd414fb44cebc2bf56b431
I20240503 14:06:04.578166 18407 raft_consensus.cc:2781] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:06:04.576784 18403 ts_tablet_manager.cc:1398] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1: Time spent starting tablet: real 0.027s user 0.005s sys 0.002s
I20240503 14:06:04.583532 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2540f6226da9468195f0ded09c888a6b" candidate_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fede8e81a5df42febf80ef0911d772d1" is_pre_election: true
I20240503 14:06:04.584013 18406 tablet_bootstrap.cc:492] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431: No bootstrap required, opened a new log
I20240503 14:06:04.584283 18284 raft_consensus.cc:2454] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c138f5b79ac428886fcaeb9ff725dd9 in term 0.
I20240503 14:06:04.584568 18406 ts_tablet_manager.cc:1367] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431: Time spent bootstrapping tablet: real 0.061s user 0.017s sys 0.003s
I20240503 14:06:04.585794 18172 leader_election.cc:302] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c138f5b79ac428886fcaeb9ff725dd9, fede8e81a5df42febf80ef0911d772d1; no voters:
I20240503 14:06:04.587002 18411 raft_consensus.cc:2781] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:06:04.587633 18411 raft_consensus.cc:482] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:04.587994 18411 raft_consensus.cc:3037] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:04.589147 18208 raft_consensus.cc:2454] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43858e199ddd414fb44cebc2bf56b431 in term 1.
I20240503 14:06:04.590581 18321 leader_election.cc:302] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43858e199ddd414fb44cebc2bf56b431, 5c138f5b79ac428886fcaeb9ff725dd9; no voters:
I20240503 14:06:04.591495 18412 raft_consensus.cc:2781] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:06:04.587095 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c372eadbe3de4dda8c100977d1f017a7" candidate_uuid: "43858e199ddd414fb44cebc2bf56b431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fede8e81a5df42febf80ef0911d772d1"
I20240503 14:06:04.593254 18284 raft_consensus.cc:3037] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:04.593250 18406 raft_consensus.cc:348] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.594401 18406 raft_consensus.cc:374] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:04.594736 18406 raft_consensus.cc:729] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43858e199ddd414fb44cebc2bf56b431, State: Initialized, Role: FOLLOWER
I20240503 14:06:04.595717 18406 consensus_queue.cc:260] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.599170 18407 raft_consensus.cc:686] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 1 LEADER]: Becoming Leader. State: Replica: fede8e81a5df42febf80ef0911d772d1, State: Running, Role: LEADER
I20240503 14:06:04.606364 18284 raft_consensus.cc:2454] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43858e199ddd414fb44cebc2bf56b431 in term 1.
I20240503 14:06:04.610037 18407 consensus_queue.cc:237] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.607573 18412 raft_consensus.cc:686] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 1 LEADER]: Becoming Leader. State: Replica: 43858e199ddd414fb44cebc2bf56b431, State: Running, Role: LEADER
I20240503 14:06:04.612569 18411 raft_consensus.cc:504] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.616441 18106 catalog_manager.cc:5484] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fede8e81a5df42febf80ef0911d772d1 (127.13.158.66). New cstate: current_term: 1 leader_uuid: "fede8e81a5df42febf80ef0911d772d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: UNKNOWN } } }
I20240503 14:06:04.618702 18406 ts_tablet_manager.cc:1398] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431: Time spent starting tablet: real 0.034s user 0.009s sys 0.009s
I20240503 14:06:04.618880 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2540f6226da9468195f0ded09c888a6b" candidate_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43858e199ddd414fb44cebc2bf56b431" is_pre_election: true
I20240503 14:06:04.619940 18358 raft_consensus.cc:2454] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c138f5b79ac428886fcaeb9ff725dd9 in term 0.
I20240503 14:06:04.612749 18412 consensus_queue.cc:237] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.621745 18411 leader_election.cc:288] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [CANDIDATE]: Term 1 election: Requested vote from peers fede8e81a5df42febf80ef0911d772d1 (127.13.158.66:41189), 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67:34431)
I20240503 14:06:04.623031 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2540f6226da9468195f0ded09c888a6b" candidate_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fede8e81a5df42febf80ef0911d772d1"
I20240503 14:06:04.623171 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2540f6226da9468195f0ded09c888a6b" candidate_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43858e199ddd414fb44cebc2bf56b431"
I20240503 14:06:04.623705 18284 raft_consensus.cc:3037] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:04.623828 18358 raft_consensus.cc:3037] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:04.631079 18284 raft_consensus.cc:2454] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c138f5b79ac428886fcaeb9ff725dd9 in term 1.
I20240503 14:06:04.632285 18358 raft_consensus.cc:2454] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c138f5b79ac428886fcaeb9ff725dd9 in term 1.
I20240503 14:06:04.632477 18172 leader_election.cc:302] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c138f5b79ac428886fcaeb9ff725dd9, fede8e81a5df42febf80ef0911d772d1; no voters:
I20240503 14:06:04.633251 18411 raft_consensus.cc:2781] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:06:04.634918 18411 raft_consensus.cc:686] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c138f5b79ac428886fcaeb9ff725dd9, State: Running, Role: LEADER
I20240503 14:06:04.635936 18411 consensus_queue.cc:237] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.645752 18107 catalog_manager.cc:5484] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67). New cstate: current_term: 1 leader_uuid: "43858e199ddd414fb44cebc2bf56b431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: HEALTHY } } }
I20240503 14:06:04.648336 18103 catalog_manager.cc:5484] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65). New cstate: current_term: 1 leader_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: UNKNOWN } } }
I20240503 14:06:04.655378 18106 catalog_manager.cc:5484] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fede8e81a5df42febf80ef0911d772d1 (127.13.158.66). New cstate: current_term: 1 leader_uuid: "fede8e81a5df42febf80ef0911d772d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: UNKNOWN } } }
I20240503 14:06:04.675552 13945 tablet_server.cc:178] TabletServer@127.13.158.65:0 shutting down...
I20240503 14:06:04.695343 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:06:04.696121 13945 tablet_replica.cc:330] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9: stopping tablet replica
I20240503 14:06:04.696736 13945 raft_consensus.cc:2229] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:06:04.697310 13945 pending_rounds.cc:62] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9: Trying to abort 1 pending ops.
I20240503 14:06:04.697530 13945 pending_rounds.cc:68] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023596194358685696 op_type: NO_OP noop_request { }
I20240503 14:06:04.697836 13945 raft_consensus.cc:2866] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:06:04.698096 13945 raft_consensus.cc:2258] T 2540f6226da9468195f0ded09c888a6b P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:04.700232 13945 tablet_replica.cc:330] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9: stopping tablet replica
I20240503 14:06:04.700752 13945 raft_consensus.cc:2229] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:04.701134 13945 raft_consensus.cc:2258] T 2f053f9a1b9e41f68fc778572aaf64cf P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:04.703135 13945 tablet_replica.cc:330] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9: stopping tablet replica
I20240503 14:06:04.703636 13945 raft_consensus.cc:2229] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:04.703979 13945 raft_consensus.cc:2258] T c372eadbe3de4dda8c100977d1f017a7 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:04.705746 13945 tablet_replica.cc:330] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9: stopping tablet replica
I20240503 14:06:04.706209 13945 raft_consensus.cc:2229] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:04.706598 13945 raft_consensus.cc:2258] T 4e478e48cb48476e9cde5fd215be0e02 P 5c138f5b79ac428886fcaeb9ff725dd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:04.728753 13945 tablet_server.cc:194] TabletServer@127.13.158.65:0 shutdown complete.
I20240503 14:06:04.759601 18412 raft_consensus.cc:482] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:04.760121 18412 raft_consensus.cc:504] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:04.761653 18412 leader_election.cc:288] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717), fede8e81a5df42febf80ef0911d772d1 (127.13.158.66:41189)
W20240503 14:06:04.762931 18321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111) [suppressed 5 similar messages]
I20240503 14:06:04.763267 18284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f053f9a1b9e41f68fc778572aaf64cf" candidate_uuid: "43858e199ddd414fb44cebc2bf56b431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fede8e81a5df42febf80ef0911d772d1" is_pre_election: true
W20240503 14:06:04.766216 18321 leader_election.cc:334] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111)
I20240503 14:06:04.766613 18321 leader_election.cc:302] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43858e199ddd414fb44cebc2bf56b431; no voters: 5c138f5b79ac428886fcaeb9ff725dd9, fede8e81a5df42febf80ef0911d772d1
I20240503 14:06:04.767207 18412 raft_consensus.cc:2726] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:06:04.999533 18247 consensus_peers.cc:485] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 -> Peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Couldn't send request to peer 5c138f5b79ac428886fcaeb9ff725dd9. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:05.083189 18407 consensus_queue.cc:1035] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:06:05.106333 18412 consensus_queue.cc:1035] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:06:05.108457 18247 consensus_peers.cc:485] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 -> Peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Couldn't send request to peer 5c138f5b79ac428886fcaeb9ff725dd9. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:06:05.117758 18321 consensus_peers.cc:485] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 -> Peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Couldn't send request to peer 5c138f5b79ac428886fcaeb9ff725dd9. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:05.203408 18358 raft_consensus.cc:3037] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:05.209626 18407 consensus_queue.cc:1035] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:06:05.666935 18412 consensus_queue.cc:579] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [LEADER]: Leader has been unable to successfully communicate with peer 5c138f5b79ac428886fcaeb9ff725dd9 for more than 1 seconds (1.044s)
I20240503 14:06:05.672827 18428 consensus_queue.cc:579] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Leader has been unable to successfully communicate with peer 5c138f5b79ac428886fcaeb9ff725dd9 for more than 1 seconds (1.054s)
W20240503 14:06:05.675333 18103 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c372eadbe3de4dda8c100977d1f017a7 with cas_config_opid_index -1: no extra replica candidate found for tablet c372eadbe3de4dda8c100977d1f017a7 (table test-workload [id=2beb719d24b14d4cb9e866bcac57bede]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20240503 14:06:05.741034 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:06:05.748531 18447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:05.749415 18448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:05.750092 18450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:05.750936 13945 server_base.cc:1034] running on GCE node
I20240503 14:06:05.752337 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:05.752552 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:05.752708 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745165752690 us; error 0 us; skew 500 ppm
I20240503 14:06:05.753234 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:05.760979 13945 webserver.cc:460] Webserver started at http://127.13.158.68:42653/ using document root <none> and password file <none>
I20240503 14:06:05.761512 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:05.761708 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:06:05.762017 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:06:05.763188 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-3-root/instance:
uuid: "aa8528c7ad36445981120796c198d840"
format_stamp: "Formatted at 2024-05-03 14:06:05 on dist-test-slave-jm3r"
I20240503 14:06:05.767697 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
W20240503 14:06:05.771332 18155 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 5c138f5b79ac428886fcaeb9ff725dd9 not available for placement
I20240503 14:06:05.771597 18455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:05.772459 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:06:05.772828 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-3-root
uuid: "aa8528c7ad36445981120796c198d840"
format_stamp: "Formatted at 2024-05-03 14:06:05 on dist-test-slave-jm3r"
I20240503 14:06:05.773180 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714745036514908-13945-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:05.783502 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:05.784446 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:05.786171 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:06:05.789209 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:06:05.789403 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:05.789600 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:06:05.789724 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:05.834110 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.68:34747
I20240503 14:06:05.834206 18517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.68:34747 every 8 connection(s)
I20240503 14:06:05.848498 18518 heartbeater.cc:347] Connected to a master server at 127.13.158.126:40425
I20240503 14:06:05.848852 18518 heartbeater.cc:464] Registering TS with master...
I20240503 14:06:05.849591 18518 heartbeater.cc:511] Master 127.13.158.126:40425 requested a full tablet report, sending...
I20240503 14:06:05.851392 18106 ts_manager.cc:194] Registered new tserver with Master: aa8528c7ad36445981120796c198d840 (127.13.158.68:34747)
I20240503 14:06:05.852838 18106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47840
I20240503 14:06:05.868855 18358 consensus_queue.cc:237] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } }
I20240503 14:06:05.874907 18284 raft_consensus.cc:1261] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Refusing update from remote peer 43858e199ddd414fb44cebc2bf56b431: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:06:05.877025 18443 consensus_queue.cc:1035] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:06:05.879918 18321 consensus_peers.cc:485] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 -> Peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Couldn't send request to peer 5c138f5b79ac428886fcaeb9ff725dd9. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:05.885114 18444 raft_consensus.cc:2932] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aa8528c7ad36445981120796c198d840 (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } } }
I20240503 14:06:05.887032 18284 raft_consensus.cc:2932] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aa8528c7ad36445981120796c198d840 (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } } }
W20240503 14:06:05.888491 18322 consensus_peers.cc:485] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 -> Peer aa8528c7ad36445981120796c198d840 (127.13.158.68:34747): Couldn't send request to peer aa8528c7ad36445981120796c198d840. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c372eadbe3de4dda8c100977d1f017a7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:05.893304 18093 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c372eadbe3de4dda8c100977d1f017a7 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20240503 14:06:05.897104 18106 catalog_manager.cc:5484] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 reported cstate change: config changed from index -1 to 2, NON_VOTER aa8528c7ad36445981120796c198d840 (127.13.158.68) added. New cstate: current_term: 1 leader_uuid: "43858e199ddd414fb44cebc2bf56b431" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:06:05.954160 18428 consensus_queue.cc:579] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Leader has been unable to successfully communicate with peer 5c138f5b79ac428886fcaeb9ff725dd9 for more than 1 seconds (1.385s)
I20240503 14:06:05.962934 18284 consensus_queue.cc:237] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } }
I20240503 14:06:05.968683 18358 raft_consensus.cc:1261] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:06:05.970788 18407 consensus_queue.cc:1035] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240503 14:06:05.973893 18247 consensus_peers.cc:485] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 -> Peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Couldn't send request to peer 5c138f5b79ac428886fcaeb9ff725dd9. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:06:05.978663 18248 consensus_peers.cc:485] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 -> Peer aa8528c7ad36445981120796c198d840 (127.13.158.68:34747): Couldn't send request to peer aa8528c7ad36445981120796c198d840. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e478e48cb48476e9cde5fd215be0e02. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:05.978173 18428 raft_consensus.cc:2932] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aa8528c7ad36445981120796c198d840 (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } } }
I20240503 14:06:05.980134 18358 raft_consensus.cc:2932] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aa8528c7ad36445981120796c198d840 (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } } }
I20240503 14:06:05.986058 18091 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4e478e48cb48476e9cde5fd215be0e02 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:06:05.990207 18103 catalog_manager.cc:5484] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 reported cstate change: config changed from index -1 to 2, NON_VOTER aa8528c7ad36445981120796c198d840 (127.13.158.68) added. New cstate: current_term: 1 leader_uuid: "fede8e81a5df42febf80ef0911d772d1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:06:06.008631 18284 consensus_queue.cc:237] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } }
I20240503 14:06:06.013777 18358 raft_consensus.cc:1261] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:06:06.014065 18248 consensus_peers.cc:485] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 -> Peer aa8528c7ad36445981120796c198d840 (127.13.158.68:34747): Couldn't send request to peer aa8528c7ad36445981120796c198d840. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f053f9a1b9e41f68fc778572aaf64cf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:06.015080 18428 consensus_queue.cc:1035] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:06:06.016597 18247 consensus_peers.cc:485] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 -> Peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Couldn't send request to peer 5c138f5b79ac428886fcaeb9ff725dd9. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:06.021186 18423 raft_consensus.cc:2932] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aa8528c7ad36445981120796c198d840 (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } } }
I20240503 14:06:06.023231 18358 raft_consensus.cc:2932] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aa8528c7ad36445981120796c198d840 (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } } }
I20240503 14:06:06.029546 18091 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2f053f9a1b9e41f68fc778572aaf64cf with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20240503 14:06:06.034970 18106 catalog_manager.cc:5484] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 reported cstate change: config changed from index -1 to 2, NON_VOTER aa8528c7ad36445981120796c198d840 (127.13.158.68) added. New cstate: current_term: 1 leader_uuid: "fede8e81a5df42febf80ef0911d772d1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:06:06.403332 18532 ts_tablet_manager.cc:902] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840: Initiating tablet copy from peer 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67:34431)
I20240503 14:06:06.406083 18532 tablet_copy_client.cc:308] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840: tablet copy: Beginning tablet copy session from remote peer at address 127.13.158.67:34431
I20240503 14:06:06.414814 18539 ts_tablet_manager.cc:902] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840: Initiating tablet copy from peer fede8e81a5df42febf80ef0911d772d1 (127.13.158.66:41189)
I20240503 14:06:06.417234 18368 tablet_copy_service.cc:140] P 43858e199ddd414fb44cebc2bf56b431: Received BeginTabletCopySession request for tablet c372eadbe3de4dda8c100977d1f017a7 from peer aa8528c7ad36445981120796c198d840 ({username='slave'} at 127.0.0.1:56180)
I20240503 14:06:06.417827 18368 tablet_copy_service.cc:161] P 43858e199ddd414fb44cebc2bf56b431: Beginning new tablet copy session on tablet c372eadbe3de4dda8c100977d1f017a7 from peer aa8528c7ad36445981120796c198d840 at {username='slave'} at 127.0.0.1:56180: session id = aa8528c7ad36445981120796c198d840-c372eadbe3de4dda8c100977d1f017a7
I20240503 14:06:06.419714 18539 tablet_copy_client.cc:308] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840: tablet copy: Beginning tablet copy session from remote peer at address 127.13.158.66:41189
I20240503 14:06:06.425392 18368 tablet_copy_source_session.cc:205] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:06:06.428894 18532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c372eadbe3de4dda8c100977d1f017a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:06.429801 18294 tablet_copy_service.cc:140] P fede8e81a5df42febf80ef0911d772d1: Received BeginTabletCopySession request for tablet 2f053f9a1b9e41f68fc778572aaf64cf from peer aa8528c7ad36445981120796c198d840 ({username='slave'} at 127.0.0.1:42592)
I20240503 14:06:06.430315 18294 tablet_copy_service.cc:161] P fede8e81a5df42febf80ef0911d772d1: Beginning new tablet copy session on tablet 2f053f9a1b9e41f68fc778572aaf64cf from peer aa8528c7ad36445981120796c198d840 at {username='slave'} at 127.0.0.1:42592: session id = aa8528c7ad36445981120796c198d840-2f053f9a1b9e41f68fc778572aaf64cf
I20240503 14:06:06.436221 18294 tablet_copy_source_session.cc:205] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:06:06.439164 18539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f053f9a1b9e41f68fc778572aaf64cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:06.444133 18532 tablet_copy_client.cc:783] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840: tablet copy: Starting download of 0 data blocks...
I20240503 14:06:06.444729 18532 tablet_copy_client.cc:647] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840: tablet copy: Starting download of 1 WAL segments...
I20240503 14:06:06.448464 18532 tablet_copy_client.cc:523] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:06:06.454190 18539 tablet_copy_client.cc:783] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840: tablet copy: Starting download of 0 data blocks...
I20240503 14:06:06.454742 18539 tablet_copy_client.cc:647] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840: tablet copy: Starting download of 1 WAL segments...
I20240503 14:06:06.455579 18532 tablet_bootstrap.cc:492] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840: Bootstrap starting.
I20240503 14:06:06.458174 18539 tablet_copy_client.cc:523] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:06:06.467869 18539 tablet_bootstrap.cc:492] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840: Bootstrap starting.
I20240503 14:06:06.468433 18545 ts_tablet_manager.cc:902] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840: Initiating tablet copy from peer fede8e81a5df42febf80ef0911d772d1 (127.13.158.66:41189)
I20240503 14:06:06.473361 18545 tablet_copy_client.cc:308] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840: tablet copy: Beginning tablet copy session from remote peer at address 127.13.158.66:41189
I20240503 14:06:06.474494 18532 tablet_bootstrap.cc:492] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:06.475368 18532 tablet_bootstrap.cc:492] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840: Bootstrap complete.
I20240503 14:06:06.475229 18294 tablet_copy_service.cc:140] P fede8e81a5df42febf80ef0911d772d1: Received BeginTabletCopySession request for tablet 4e478e48cb48476e9cde5fd215be0e02 from peer aa8528c7ad36445981120796c198d840 ({username='slave'} at 127.0.0.1:42592)
I20240503 14:06:06.475970 18294 tablet_copy_service.cc:161] P fede8e81a5df42febf80ef0911d772d1: Beginning new tablet copy session on tablet 4e478e48cb48476e9cde5fd215be0e02 from peer aa8528c7ad36445981120796c198d840 at {username='slave'} at 127.0.0.1:42592: session id = aa8528c7ad36445981120796c198d840-4e478e48cb48476e9cde5fd215be0e02
I20240503 14:06:06.476130 18532 ts_tablet_manager.cc:1367] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20240503 14:06:06.479045 18532 raft_consensus.cc:348] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } }
I20240503 14:06:06.479691 18532 raft_consensus.cc:729] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa8528c7ad36445981120796c198d840, State: Initialized, Role: LEARNER
I20240503 14:06:06.480427 18532 consensus_queue.cc:260] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } }
I20240503 14:06:06.483687 18518 heartbeater.cc:503] Master 127.13.158.126:40425 was elected leader, sending a full tablet report...
I20240503 14:06:06.484053 18532 ts_tablet_manager.cc:1398] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840: Time spent starting tablet: real 0.008s user 0.000s sys 0.007s
I20240503 14:06:06.484186 18294 tablet_copy_source_session.cc:205] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:06:06.485635 18368 tablet_copy_service.cc:342] P 43858e199ddd414fb44cebc2bf56b431: Request end of tablet copy session aa8528c7ad36445981120796c198d840-c372eadbe3de4dda8c100977d1f017a7 received from {username='slave'} at 127.0.0.1:56180
I20240503 14:06:06.485953 18368 tablet_copy_service.cc:433] P 43858e199ddd414fb44cebc2bf56b431: ending tablet copy session aa8528c7ad36445981120796c198d840-c372eadbe3de4dda8c100977d1f017a7 on tablet c372eadbe3de4dda8c100977d1f017a7 with peer aa8528c7ad36445981120796c198d840
I20240503 14:06:06.488417 18545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e478e48cb48476e9cde5fd215be0e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:06.489678 18539 tablet_bootstrap.cc:492] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:06.490563 18539 tablet_bootstrap.cc:492] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840: Bootstrap complete.
I20240503 14:06:06.491318 18539 ts_tablet_manager.cc:1367] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.003s
I20240503 14:06:06.493921 18539 raft_consensus.cc:348] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } }
I20240503 14:06:06.494799 18539 raft_consensus.cc:729] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa8528c7ad36445981120796c198d840, State: Initialized, Role: LEARNER
I20240503 14:06:06.495414 18539 consensus_queue.cc:260] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } }
I20240503 14:06:06.497591 18539 ts_tablet_manager.cc:1398] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:06:06.499413 18294 tablet_copy_service.cc:342] P fede8e81a5df42febf80ef0911d772d1: Request end of tablet copy session aa8528c7ad36445981120796c198d840-2f053f9a1b9e41f68fc778572aaf64cf received from {username='slave'} at 127.0.0.1:42592
I20240503 14:06:06.499807 18294 tablet_copy_service.cc:433] P fede8e81a5df42febf80ef0911d772d1: ending tablet copy session aa8528c7ad36445981120796c198d840-2f053f9a1b9e41f68fc778572aaf64cf on tablet 2f053f9a1b9e41f68fc778572aaf64cf with peer aa8528c7ad36445981120796c198d840
I20240503 14:06:06.504954 18545 tablet_copy_client.cc:783] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840: tablet copy: Starting download of 0 data blocks...
I20240503 14:06:06.505519 18545 tablet_copy_client.cc:647] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840: tablet copy: Starting download of 1 WAL segments...
I20240503 14:06:06.508652 18545 tablet_copy_client.cc:523] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:06:06.515417 18545 tablet_bootstrap.cc:492] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840: Bootstrap starting.
I20240503 14:06:06.528887 18545 tablet_bootstrap.cc:492] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:06.529590 18545 tablet_bootstrap.cc:492] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840: Bootstrap complete.
I20240503 14:06:06.530333 18545 ts_tablet_manager.cc:1367] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20240503 14:06:06.532982 18545 raft_consensus.cc:348] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } }
I20240503 14:06:06.533592 18545 raft_consensus.cc:729] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa8528c7ad36445981120796c198d840, State: Initialized, Role: LEARNER
I20240503 14:06:06.534099 18545 consensus_queue.cc:260] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } }
I20240503 14:06:06.535297 18407 raft_consensus.cc:482] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:06.536106 18545 ts_tablet_manager.cc:1398] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:06:06.535822 18407 raft_consensus.cc:504] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:06.537806 18407 leader_election.cc:288] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717), 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67:34431)
I20240503 14:06:06.538811 18294 tablet_copy_service.cc:342] P fede8e81a5df42febf80ef0911d772d1: Request end of tablet copy session aa8528c7ad36445981120796c198d840-4e478e48cb48476e9cde5fd215be0e02 received from {username='slave'} at 127.0.0.1:42592
I20240503 14:06:06.539186 18294 tablet_copy_service.cc:433] P fede8e81a5df42febf80ef0911d772d1: ending tablet copy session aa8528c7ad36445981120796c198d840-4e478e48cb48476e9cde5fd215be0e02 on tablet 4e478e48cb48476e9cde5fd215be0e02 with peer aa8528c7ad36445981120796c198d840
I20240503 14:06:06.538964 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2540f6226da9468195f0ded09c888a6b" candidate_uuid: "fede8e81a5df42febf80ef0911d772d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43858e199ddd414fb44cebc2bf56b431" is_pre_election: true
I20240503 14:06:06.539734 18358 raft_consensus.cc:2454] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fede8e81a5df42febf80ef0911d772d1 in term 1.
I20240503 14:06:06.541731 18249 leader_election.cc:302] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43858e199ddd414fb44cebc2bf56b431, fede8e81a5df42febf80ef0911d772d1; no voters:
W20240503 14:06:06.543547 18247 leader_election.cc:334] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111)
I20240503 14:06:06.543768 18407 raft_consensus.cc:2781] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:06:06.544179 18407 raft_consensus.cc:482] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:06.544534 18407 raft_consensus.cc:3037] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:06.550103 18407 raft_consensus.cc:504] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:06.551767 18407 leader_election.cc:288] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 2 election: Requested vote from peers 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717), 43858e199ddd414fb44cebc2bf56b431 (127.13.158.67:34431)
I20240503 14:06:06.552852 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2540f6226da9468195f0ded09c888a6b" candidate_uuid: "fede8e81a5df42febf80ef0911d772d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43858e199ddd414fb44cebc2bf56b431"
I20240503 14:06:06.553483 18358 raft_consensus.cc:3037] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:06:06.555212 18247 leader_election.cc:334] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111)
I20240503 14:06:06.558825 18358 raft_consensus.cc:2454] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fede8e81a5df42febf80ef0911d772d1 in term 2.
I20240503 14:06:06.559826 18249 leader_election.cc:302] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43858e199ddd414fb44cebc2bf56b431, fede8e81a5df42febf80ef0911d772d1; no voters: 5c138f5b79ac428886fcaeb9ff725dd9
I20240503 14:06:06.560591 18407 raft_consensus.cc:2781] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:06:06.560967 18407 raft_consensus.cc:686] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 2 LEADER]: Becoming Leader. State: Replica: fede8e81a5df42febf80ef0911d772d1, State: Running, Role: LEADER
I20240503 14:06:06.561739 18407 consensus_queue.cc:237] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } }
I20240503 14:06:06.567934 18103 catalog_manager.cc:5484] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 reported cstate change: term changed from 1 to 2, leader changed from 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) to fede8e81a5df42febf80ef0911d772d1 (127.13.158.66). New cstate: current_term: 2 leader_uuid: "fede8e81a5df42febf80ef0911d772d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: UNKNOWN } } }
W20240503 14:06:06.773291 18155 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 5c138f5b79ac428886fcaeb9ff725dd9 not available for placement
I20240503 14:06:06.827663 18493 raft_consensus.cc:1203] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:06:06.869549 18493 raft_consensus.cc:1203] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:06:06.945475 18358 raft_consensus.cc:1261] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 2 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 14:06:06.946833 18558 consensus_queue.cc:1035] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:06:06.963186 18247 consensus_peers.cc:485] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 -> Peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Couldn't send request to peer 5c138f5b79ac428886fcaeb9ff725dd9. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:06.979488 18493 raft_consensus.cc:1203] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:06:07.019054 18407 consensus_queue.cc:579] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Leader has been unable to successfully communicate with peer 5c138f5b79ac428886fcaeb9ff725dd9 for more than 1 seconds (1.009s)
I20240503 14:06:07.305734 18407 raft_consensus.cc:1050] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1: attempting to promote NON_VOTER aa8528c7ad36445981120796c198d840 to VOTER
I20240503 14:06:07.308641 18407 consensus_queue.cc:237] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } }
I20240503 14:06:07.317117 18493 raft_consensus.cc:1261] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840 [term 1 LEARNER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:06:07.317739 18358 raft_consensus.cc:1261] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:06:07.318814 18558 consensus_queue.cc:1035] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:06:07.319798 18407 consensus_queue.cc:1035] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 14:06:07.321542 18247 consensus_peers.cc:485] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 -> Peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Couldn't send request to peer 5c138f5b79ac428886fcaeb9ff725dd9. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:07.328896 18407 raft_consensus.cc:2932] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.330597 18493 raft_consensus.cc:2932] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.342976 18358 raft_consensus.cc:2932] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.347360 18106 catalog_manager.cc:5484] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840 reported cstate change: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fede8e81a5df42febf80ef0911d772d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.350286 18443 consensus_queue.cc:579] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [LEADER]: Leader has been unable to successfully communicate with peer 5c138f5b79ac428886fcaeb9ff725dd9 for more than 1 seconds (1.480s)
I20240503 14:06:07.376850 18284 consensus_queue.cc:237] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } }
I20240503 14:06:07.381681 18493 raft_consensus.cc:1261] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:06:07.382167 18358 raft_consensus.cc:1261] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:06:07.383697 18423 consensus_queue.cc:1035] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:06:07.384542 18407 consensus_queue.cc:1035] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:06:07.392477 18443 raft_consensus.cc:1050] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431: attempting to promote NON_VOTER aa8528c7ad36445981120796c198d840 to VOTER
I20240503 14:06:07.395560 18443 consensus_queue.cc:237] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } }
I20240503 14:06:07.397543 18493 raft_consensus.cc:2932] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.397760 18407 raft_consensus.cc:2932] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.400012 18358 raft_consensus.cc:2932] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.411648 18284 raft_consensus.cc:1261] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Refusing update from remote peer 43858e199ddd414fb44cebc2bf56b431: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:06:07.412915 18091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f053f9a1b9e41f68fc778572aaf64cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:06:07.413942 18493 raft_consensus.cc:1261] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840 [term 1 LEARNER]: Refusing update from remote peer 43858e199ddd414fb44cebc2bf56b431: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:06:07.414271 18560 consensus_queue.cc:1035] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:06:07.416023 18559 consensus_queue.cc:1035] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:06:07.413609 18107 catalog_manager.cc:5484] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840 reported cstate change: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New cstate: current_term: 1 leader_uuid: "fede8e81a5df42febf80ef0911d772d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
W20240503 14:06:07.426095 18321 consensus_peers.cc:485] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 -> Peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Couldn't send request to peer 5c138f5b79ac428886fcaeb9ff725dd9. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:07.432569 18443 raft_consensus.cc:2932] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
W20240503 14:06:07.439110 18091 catalog_manager.cc:4573] TS 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2f053f9a1b9e41f68fc778572aaf64cf: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111)
I20240503 14:06:07.438326 18493 raft_consensus.cc:2932] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.445932 18284 raft_consensus.cc:2932] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.450630 18407 consensus_queue.cc:579] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Leader has been unable to successfully communicate with peer 5c138f5b79ac428886fcaeb9ff725dd9 for more than 1 seconds (1.487s)
I20240503 14:06:07.450564 18106 catalog_manager.cc:5484] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 reported cstate change: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43858e199ddd414fb44cebc2bf56b431" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:06:07.466053 18358 consensus_queue.cc:237] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } }
I20240503 14:06:07.471788 18493 raft_consensus.cc:1261] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Refusing update from remote peer 43858e199ddd414fb44cebc2bf56b431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:06:07.472329 18284 raft_consensus.cc:1261] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Refusing update from remote peer 43858e199ddd414fb44cebc2bf56b431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:06:07.473067 18443 consensus_queue.cc:1035] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:06:07.473846 18559 consensus_queue.cc:1035] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:06:07.480276 18560 raft_consensus.cc:2932] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.482162 18493 raft_consensus.cc:2932] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.482339 18284 raft_consensus.cc:2932] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.488572 18093 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c372eadbe3de4dda8c100977d1f017a7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:06:07.493005 18106 catalog_manager.cc:5484] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 reported cstate change: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New cstate: current_term: 1 leader_uuid: "43858e199ddd414fb44cebc2bf56b431" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:06:07.502166 18558 raft_consensus.cc:1050] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1: attempting to promote NON_VOTER aa8528c7ad36445981120796c198d840 to VOTER
I20240503 14:06:07.505040 18558 consensus_queue.cc:237] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } }
I20240503 14:06:07.512655 18493 raft_consensus.cc:1261] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840 [term 1 LEARNER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:06:07.514314 18579 consensus_queue.cc:1035] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:06:07.514753 18358 raft_consensus.cc:1261] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240503 14:06:07.515568 18247 consensus_peers.cc:485] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 -> Peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Couldn't send request to peer 5c138f5b79ac428886fcaeb9ff725dd9. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:07.516454 18579 consensus_queue.cc:1035] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:06:07.523821 18558 raft_consensus.cc:2932] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.525615 18358 raft_consensus.cc:2932] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.536742 18493 raft_consensus.cc:2932] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.538727 18106 catalog_manager.cc:5484] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 reported cstate change: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fede8e81a5df42febf80ef0911d772d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:06:07.551726 18284 consensus_queue.cc:237] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } }
I20240503 14:06:07.556447 18358 raft_consensus.cc:1261] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:06:07.556447 18493 raft_consensus.cc:1261] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:06:07.558136 18407 consensus_queue.cc:1035] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:06:07.559031 18558 consensus_queue.cc:1035] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:06:07.565795 18579 raft_consensus.cc:2932] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.567438 18493 raft_consensus.cc:2932] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.567526 18358 raft_consensus.cc:2932] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:07.576589 18091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e478e48cb48476e9cde5fd215be0e02 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:06:07.578961 18107 catalog_manager.cc:5484] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840 reported cstate change: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New cstate: current_term: 1 leader_uuid: "fede8e81a5df42febf80ef0911d772d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
W20240503 14:06:07.776583 18155 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 5c138f5b79ac428886fcaeb9ff725dd9 not available for placement
I20240503 14:06:07.923030 18558 consensus_queue.cc:579] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Leader has been unable to successfully communicate with peer 5c138f5b79ac428886fcaeb9ff725dd9 for more than 1 seconds (1.360s)
I20240503 14:06:07.933907 18284 consensus_queue.cc:237] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } }
I20240503 14:06:07.939451 18358 raft_consensus.cc:1261] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 2 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240503 14:06:07.940043 18248 consensus_peers.cc:485] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 -> Peer aa8528c7ad36445981120796c198d840 (127.13.158.68:34747): Couldn't send request to peer aa8528c7ad36445981120796c198d840. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2540f6226da9468195f0ded09c888a6b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:07.940958 18558 consensus_queue.cc:1035] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:06:07.942159 18247 consensus_peers.cc:485] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 -> Peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Couldn't send request to peer 5c138f5b79ac428886fcaeb9ff725dd9. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:07.947716 18423 raft_consensus.cc:2932] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER aa8528c7ad36445981120796c198d840 (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } } }
I20240503 14:06:07.949575 18358 raft_consensus.cc:2932] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER aa8528c7ad36445981120796c198d840 (127.13.158.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } } }
I20240503 14:06:07.955675 18091 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2540f6226da9468195f0ded09c888a6b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:06:07.959992 18106 catalog_manager.cc:5484] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 reported cstate change: config changed from index -1 to 2, NON_VOTER aa8528c7ad36445981120796c198d840 (127.13.158.68) added. New cstate: current_term: 2 leader_uuid: "fede8e81a5df42febf80ef0911d772d1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } } }
I20240503 14:06:08.533538 18584 ts_tablet_manager.cc:902] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840: Initiating tablet copy from peer fede8e81a5df42febf80ef0911d772d1 (127.13.158.66:41189)
I20240503 14:06:08.536456 18584 tablet_copy_client.cc:308] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840: tablet copy: Beginning tablet copy session from remote peer at address 127.13.158.66:41189
I20240503 14:06:08.538172 18294 tablet_copy_service.cc:140] P fede8e81a5df42febf80ef0911d772d1: Received BeginTabletCopySession request for tablet 2540f6226da9468195f0ded09c888a6b from peer aa8528c7ad36445981120796c198d840 ({username='slave'} at 127.0.0.1:42592)
I20240503 14:06:08.538678 18294 tablet_copy_service.cc:161] P fede8e81a5df42febf80ef0911d772d1: Beginning new tablet copy session on tablet 2540f6226da9468195f0ded09c888a6b from peer aa8528c7ad36445981120796c198d840 at {username='slave'} at 127.0.0.1:42592: session id = aa8528c7ad36445981120796c198d840-2540f6226da9468195f0ded09c888a6b
I20240503 14:06:08.544550 18294 tablet_copy_source_session.cc:205] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:06:08.547317 18584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2540f6226da9468195f0ded09c888a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:08.559007 18584 tablet_copy_client.cc:783] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840: tablet copy: Starting download of 0 data blocks...
I20240503 14:06:08.559681 18584 tablet_copy_client.cc:647] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840: tablet copy: Starting download of 1 WAL segments...
I20240503 14:06:08.563143 18584 tablet_copy_client.cc:523] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:06:08.569545 18584 tablet_bootstrap.cc:492] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840: Bootstrap starting.
I20240503 14:06:08.584656 18584 tablet_bootstrap.cc:492] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:08.585564 18584 tablet_bootstrap.cc:492] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840: Bootstrap complete.
I20240503 14:06:08.586102 18584 ts_tablet_manager.cc:1367] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240503 14:06:08.588110 18584 raft_consensus.cc:348] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } }
I20240503 14:06:08.588685 18584 raft_consensus.cc:729] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: aa8528c7ad36445981120796c198d840, State: Initialized, Role: LEARNER
I20240503 14:06:08.589141 18584 consensus_queue.cc:260] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: true } }
I20240503 14:06:08.591616 18584 ts_tablet_manager.cc:1398] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20240503 14:06:08.593310 18294 tablet_copy_service.cc:342] P fede8e81a5df42febf80ef0911d772d1: Request end of tablet copy session aa8528c7ad36445981120796c198d840-2540f6226da9468195f0ded09c888a6b received from {username='slave'} at 127.0.0.1:42592
I20240503 14:06:08.593739 18294 tablet_copy_service.cc:433] P fede8e81a5df42febf80ef0911d772d1: ending tablet copy session aa8528c7ad36445981120796c198d840-2540f6226da9468195f0ded09c888a6b on tablet 2540f6226da9468195f0ded09c888a6b with peer aa8528c7ad36445981120796c198d840
W20240503 14:06:08.671722 18091 catalog_manager.cc:4573] TS 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2f053f9a1b9e41f68fc778572aaf64cf: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111)
W20240503 14:06:08.779335 18155 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 5c138f5b79ac428886fcaeb9ff725dd9 not available for placement
I20240503 14:06:08.989394 18558 consensus_queue.cc:579] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Leader has been unable to successfully communicate with peer 5c138f5b79ac428886fcaeb9ff725dd9 for more than 1 seconds (1.054s)
I20240503 14:06:09.152612 18493 raft_consensus.cc:1203] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240503 14:06:09.560392 18591 raft_consensus.cc:1050] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1: attempting to promote NON_VOTER aa8528c7ad36445981120796c198d840 to VOTER
I20240503 14:06:09.562745 18591 consensus_queue.cc:237] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } }
I20240503 14:06:09.569118 18358 raft_consensus.cc:1261] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 2 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:06:09.570217 18493 raft_consensus.cc:1261] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840 [term 2 LEARNER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:06:09.570986 18592 consensus_queue.cc:1035] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:06:09.572005 18558 consensus_queue.cc:1035] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 14:06:09.575474 18247 consensus_peers.cc:485] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 -> Peer 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): Couldn't send request to peer 5c138f5b79ac428886fcaeb9ff725dd9. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:09.582890 18592 raft_consensus.cc:2932] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:09.584678 18493 raft_consensus.cc:2932] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:09.592051 18358 raft_consensus.cc:2932] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:09.594020 18106 catalog_manager.cc:5484] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 reported cstate change: config changed from index 2 to 3, aa8528c7ad36445981120796c198d840 (127.13.158.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "fede8e81a5df42febf80ef0911d772d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5c138f5b79ac428886fcaeb9ff725dd9" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 34717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:06:09.609720 18284 consensus_queue.cc:237] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } }
I20240503 14:06:09.614396 18358 raft_consensus.cc:1261] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 2 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:06:09.614468 18493 raft_consensus.cc:1261] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840 [term 2 FOLLOWER]: Refusing update from remote peer fede8e81a5df42febf80ef0911d772d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:06:09.615769 18558 consensus_queue.cc:1035] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:06:09.618351 18558 consensus_queue.cc:1035] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:06:09.621206 18591 raft_consensus.cc:2932] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:09.623149 18358 raft_consensus.cc:2932] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:09.626330 18493 raft_consensus.cc:2932] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } } }
I20240503 14:06:09.630333 18091 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2540f6226da9468195f0ded09c888a6b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:06:09.635442 18107 catalog_manager.cc:5484] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 reported cstate change: config changed from index 3 to 4, VOTER 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65) evicted. New cstate: current_term: 2 leader_uuid: "fede8e81a5df42febf80ef0911d772d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fede8e81a5df42febf80ef0911d772d1" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43858e199ddd414fb44cebc2bf56b431" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 34431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8528c7ad36445981120796c198d840" member_type: VOTER last_known_addr { host: "127.13.158.68" port: 34747 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:06:09.688697 18091 catalog_manager.cc:4573] TS 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2540f6226da9468195f0ded09c888a6b: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111)
W20240503 14:06:09.781888 18091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240503 14:06:10.815500 18091 catalog_manager.cc:4573] TS 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2540f6226da9468195f0ded09c888a6b: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111)
W20240503 14:06:11.853411 18091 catalog_manager.cc:4573] TS 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c372eadbe3de4dda8c100977d1f017a7: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111)
I20240503 14:06:12.869513 13945 tablet_server.cc:178] TabletServer@127.13.158.66:0 shutting down...
I20240503 14:06:12.895025 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:06:12.895774 13945 tablet_replica.cc:330] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1: stopping tablet replica
I20240503 14:06:12.896373 13945 raft_consensus.cc:2229] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 2 LEADER]: Raft consensus shutting down.
I20240503 14:06:12.897164 13945 raft_consensus.cc:2258] T 2540f6226da9468195f0ded09c888a6b P fede8e81a5df42febf80ef0911d772d1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:12.899350 13945 tablet_replica.cc:330] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1: stopping tablet replica
I20240503 14:06:12.899891 13945 raft_consensus.cc:2229] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:06:12.900717 13945 raft_consensus.cc:2258] T 2f053f9a1b9e41f68fc778572aaf64cf P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:12.903028 13945 tablet_replica.cc:330] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1: stopping tablet replica
I20240503 14:06:12.903568 13945 raft_consensus.cc:2229] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:12.904165 13945 raft_consensus.cc:2258] T c372eadbe3de4dda8c100977d1f017a7 P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:12.906023 13945 tablet_replica.cc:330] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1: stopping tablet replica
I20240503 14:06:12.906580 13945 raft_consensus.cc:2229] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:06:12.907339 13945 raft_consensus.cc:2258] T 4e478e48cb48476e9cde5fd215be0e02 P fede8e81a5df42febf80ef0911d772d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:12.932121 13945 tablet_server.cc:194] TabletServer@127.13.158.66:0 shutdown complete.
I20240503 14:06:12.946147 13945 tablet_server.cc:178] TabletServer@127.13.158.67:0 shutting down...
I20240503 14:06:12.975394 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:06:12.976188 13945 tablet_replica.cc:330] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431: stopping tablet replica
I20240503 14:06:12.976930 13945 raft_consensus.cc:2229] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:12.977517 13945 raft_consensus.cc:2258] T 2f053f9a1b9e41f68fc778572aaf64cf P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:12.979422 13945 tablet_replica.cc:330] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431: stopping tablet replica
I20240503 14:06:12.979954 13945 raft_consensus.cc:2229] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:12.980501 13945 raft_consensus.cc:2258] T 2540f6226da9468195f0ded09c888a6b P 43858e199ddd414fb44cebc2bf56b431 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:12.982277 13945 tablet_replica.cc:330] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431: stopping tablet replica
I20240503 14:06:12.982861 13945 raft_consensus.cc:2229] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:06:12.983611 13945 raft_consensus.cc:2258] T c372eadbe3de4dda8c100977d1f017a7 P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:12.985514 13945 tablet_replica.cc:330] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431: stopping tablet replica
I20240503 14:06:12.986009 13945 raft_consensus.cc:2229] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:12.986605 13945 raft_consensus.cc:2258] T 4e478e48cb48476e9cde5fd215be0e02 P 43858e199ddd414fb44cebc2bf56b431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:13.011857 13945 tablet_server.cc:194] TabletServer@127.13.158.67:0 shutdown complete.
I20240503 14:06:13.028192 13945 tablet_server.cc:178] TabletServer@127.13.158.68:0 shutting down...
I20240503 14:06:13.053315 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:06:13.054031 13945 tablet_replica.cc:330] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840: stopping tablet replica
I20240503 14:06:13.054646 13945 raft_consensus.cc:2229] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:13.055145 13945 raft_consensus.cc:2258] T 2540f6226da9468195f0ded09c888a6b P aa8528c7ad36445981120796c198d840 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:13.056885 13945 tablet_replica.cc:330] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840: stopping tablet replica
I20240503 14:06:13.057405 13945 raft_consensus.cc:2229] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:13.057890 13945 raft_consensus.cc:2258] T 2f053f9a1b9e41f68fc778572aaf64cf P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:13.059810 13945 tablet_replica.cc:330] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840: stopping tablet replica
I20240503 14:06:13.060299 13945 raft_consensus.cc:2229] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:13.060761 13945 raft_consensus.cc:2258] T 4e478e48cb48476e9cde5fd215be0e02 P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:13.062819 13945 tablet_replica.cc:330] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840: stopping tablet replica
I20240503 14:06:13.063279 13945 raft_consensus.cc:2229] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:13.063748 13945 raft_consensus.cc:2258] T c372eadbe3de4dda8c100977d1f017a7 P aa8528c7ad36445981120796c198d840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:13.086097 13945 tablet_server.cc:194] TabletServer@127.13.158.68:0 shutdown complete.
I20240503 14:06:13.099407 13945 master.cc:537] Master@127.13.158.126:40425 shutting down...
W20240503 14:06:13.971341 18091 catalog_manager.cc:4573] TS 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2540f6226da9468195f0ded09c888a6b: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111)
W20240503 14:06:15.927105 18091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 14:06:15.930624 18091 catalog_manager.cc:4573] TS 5c138f5b79ac428886fcaeb9ff725dd9 (127.13.158.65:34717): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2f053f9a1b9e41f68fc778572aaf64cf: Network error: Client connection negotiation failed: client connection to 127.13.158.65:34717: connect: Connection refused (error 111)
I20240503 14:06:16.713596 13945 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:06:16.714237 13945 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:16.714610 13945 tablet_replica.cc:330] T 00000000000000000000000000000000 P 34d9f7b5283843c7a77c3293ffe384a6: stopping tablet replica
I20240503 14:06:16.738303 13945 master.cc:552] Master@127.13.158.126:40425 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (14156 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20240503 14:06:16.774317 13945 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.13.158.126:42091
I20240503 14:06:16.775461 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:06:16.781800 18618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:16.782689 18619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:16.783440 18621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:16.784447 13945 server_base.cc:1034] running on GCE node
I20240503 14:06:16.785274 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:16.785465 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:16.785635 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745176785619 us; error 0 us; skew 500 ppm
I20240503 14:06:16.786135 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:16.793134 13945 webserver.cc:460] Webserver started at http://127.13.158.126:37469/ using document root <none> and password file <none>
I20240503 14:06:16.793638 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:16.793810 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:06:16.794075 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:06:16.795440 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/master-0-root/instance:
uuid: "1ec90a195d314d6f9d2354900782c557"
format_stamp: "Formatted at 2024-05-03 14:06:16 on dist-test-slave-jm3r"
I20240503 14:06:16.800046 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:06:16.803491 18626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:16.804250 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:06:16.804535 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/master-0-root
uuid: "1ec90a195d314d6f9d2354900782c557"
format_stamp: "Formatted at 2024-05-03 14:06:16 on dist-test-slave-jm3r"
I20240503 14:06:16.804829 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:16.820307 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:16.821545 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:16.857633 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.126:42091
I20240503 14:06:16.857719 18677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.126:42091 every 8 connection(s)
I20240503 14:06:16.861322 18678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:16.871739 18678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557: Bootstrap starting.
I20240503 14:06:16.875860 18678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:16.879798 18678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557: No bootstrap required, opened a new log
I20240503 14:06:16.881676 18678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec90a195d314d6f9d2354900782c557" member_type: VOTER }
I20240503 14:06:16.882045 18678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:16.882261 18678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec90a195d314d6f9d2354900782c557, State: Initialized, Role: FOLLOWER
I20240503 14:06:16.882819 18678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec90a195d314d6f9d2354900782c557" member_type: VOTER }
I20240503 14:06:16.883267 18678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:06:16.883471 18678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:06:16.883754 18678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:16.888229 18678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec90a195d314d6f9d2354900782c557" member_type: VOTER }
I20240503 14:06:16.888744 18678 leader_election.cc:302] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ec90a195d314d6f9d2354900782c557; no voters:
I20240503 14:06:16.889874 18678 leader_election.cc:288] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:06:16.890295 18681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:06:16.891541 18681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [term 1 LEADER]: Becoming Leader. State: Replica: 1ec90a195d314d6f9d2354900782c557, State: Running, Role: LEADER
I20240503 14:06:16.892185 18681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec90a195d314d6f9d2354900782c557" member_type: VOTER }
I20240503 14:06:16.892663 18678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:06:16.894924 18683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ec90a195d314d6f9d2354900782c557. Latest consensus state: current_term: 1 leader_uuid: "1ec90a195d314d6f9d2354900782c557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec90a195d314d6f9d2354900782c557" member_type: VOTER } }
I20240503 14:06:16.894874 18682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ec90a195d314d6f9d2354900782c557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec90a195d314d6f9d2354900782c557" member_type: VOTER } }
I20240503 14:06:16.895635 18683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [sys.catalog]: This master's current role is: LEADER
I20240503 14:06:16.895733 18682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557 [sys.catalog]: This master's current role is: LEADER
I20240503 14:06:16.902545 18686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:06:16.909345 18686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:06:16.910338 13945 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:06:16.917762 18686 catalog_manager.cc:1347] Generated new cluster ID: 567fda2582e94eb19ab5afd5189e6d50
I20240503 14:06:16.918081 18686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:06:16.934126 18686 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:06:16.935475 18686 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:06:16.954339 18686 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1ec90a195d314d6f9d2354900782c557: Generated new TSK 0
I20240503 14:06:16.954998 18686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:06:16.977111 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:06:16.983965 18699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:16.984922 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:16.985877 18702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:16.986958 13945 server_base.cc:1034] running on GCE node
I20240503 14:06:16.987730 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:16.987911 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:16.988075 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745176988056 us; error 0 us; skew 500 ppm
I20240503 14:06:16.988587 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:16.990929 13945 webserver.cc:460] Webserver started at http://127.13.158.65:44957/ using document root <none> and password file <none>
I20240503 14:06:16.991389 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:16.991552 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:06:16.991804 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:06:16.992941 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-0-root/instance:
uuid: "0af02be205214faaaa7f719ee14400bd"
format_stamp: "Formatted at 2024-05-03 14:06:16 on dist-test-slave-jm3r"
I20240503 14:06:16.997570 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:06:17.000960 18707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:17.001783 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:06:17.002084 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-0-root
uuid: "0af02be205214faaaa7f719ee14400bd"
format_stamp: "Formatted at 2024-05-03 14:06:16 on dist-test-slave-jm3r"
I20240503 14:06:17.002382 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:17.029487 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:17.030731 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:17.032348 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:06:17.034915 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:06:17.035118 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:17.035372 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:06:17.035534 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:17.078065 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.65:40721
I20240503 14:06:17.078158 18769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.65:40721 every 8 connection(s)
I20240503 14:06:17.082480 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:06:17.090624 18774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:17.091737 18775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:17.093653 18777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:17.094906 18770 heartbeater.cc:347] Connected to a master server at 127.13.158.126:42091
I20240503 14:06:17.094967 13945 server_base.cc:1034] running on GCE node
I20240503 14:06:17.095332 18770 heartbeater.cc:464] Registering TS with master...
I20240503 14:06:17.095932 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:17.096160 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:17.096068 18770 heartbeater.cc:511] Master 127.13.158.126:42091 requested a full tablet report, sending...
I20240503 14:06:17.096400 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745177096381 us; error 0 us; skew 500 ppm
I20240503 14:06:17.096953 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:17.098466 18643 ts_manager.cc:194] Registered new tserver with Master: 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721)
I20240503 14:06:17.099817 13945 webserver.cc:460] Webserver started at http://127.13.158.66:39891/ using document root <none> and password file <none>
I20240503 14:06:17.100437 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:17.100674 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:06:17.100962 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:06:17.100984 18643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60710
I20240503 14:06:17.102605 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-1-root/instance:
uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a"
format_stamp: "Formatted at 2024-05-03 14:06:17 on dist-test-slave-jm3r"
I20240503 14:06:17.107620 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240503 14:06:17.111100 18782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:17.111883 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240503 14:06:17.112150 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-1-root
uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a"
format_stamp: "Formatted at 2024-05-03 14:06:17 on dist-test-slave-jm3r"
I20240503 14:06:17.112414 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:17.131541 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:17.132755 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:17.134382 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:06:17.136818 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:06:17.137012 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:17.137254 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:06:17.137396 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:17.176427 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.66:38261
I20240503 14:06:17.176523 18844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.66:38261 every 8 connection(s)
I20240503 14:06:17.180954 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:06:17.211288 18850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:17.213809 18849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:17.217614 18845 heartbeater.cc:347] Connected to a master server at 127.13.158.126:42091
I20240503 14:06:17.218219 18845 heartbeater.cc:464] Registering TS with master...
W20240503 14:06:17.218246 18852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:17.219336 18845 heartbeater.cc:511] Master 127.13.158.126:42091 requested a full tablet report, sending...
I20240503 14:06:17.219393 13945 server_base.cc:1034] running on GCE node
I20240503 14:06:17.220693 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:17.220911 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:17.221100 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745177221083 us; error 0 us; skew 500 ppm
I20240503 14:06:17.221760 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:17.223105 18643 ts_manager.cc:194] Registered new tserver with Master: 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:17.225669 13945 webserver.cc:460] Webserver started at http://127.13.158.67:39445/ using document root <none> and password file <none>
I20240503 14:06:17.225775 18643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60716
I20240503 14:06:17.226547 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:17.227144 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:06:17.227728 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:06:17.229836 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-2-root/instance:
uuid: "64118e159acd4c9b89b5267d5d34ee79"
format_stamp: "Formatted at 2024-05-03 14:06:17 on dist-test-slave-jm3r"
I20240503 14:06:17.235266 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 14:06:17.239850 18857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:17.240936 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:06:17.241328 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-2-root
uuid: "64118e159acd4c9b89b5267d5d34ee79"
format_stamp: "Formatted at 2024-05-03 14:06:17 on dist-test-slave-jm3r"
I20240503 14:06:17.241649 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:17.278721 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:17.279994 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:17.281544 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:06:17.284122 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:06:17.284335 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:17.284595 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:06:17.284756 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:17.330775 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.67:37481
I20240503 14:06:17.330893 18919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.67:37481 every 8 connection(s)
I20240503 14:06:17.345476 18920 heartbeater.cc:347] Connected to a master server at 127.13.158.126:42091
I20240503 14:06:17.345834 18920 heartbeater.cc:464] Registering TS with master...
I20240503 14:06:17.346629 18920 heartbeater.cc:511] Master 127.13.158.126:42091 requested a full tablet report, sending...
I20240503 14:06:17.348459 18643 ts_manager.cc:194] Registered new tserver with Master: 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
I20240503 14:06:17.348825 13945 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014497506s
I20240503 14:06:17.350081 18643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60724
I20240503 14:06:18.104063 18770 heartbeater.cc:503] Master 127.13.158.126:42091 was elected leader, sending a full tablet report...
I20240503 14:06:18.230139 18845 heartbeater.cc:503] Master 127.13.158.126:42091 was elected leader, sending a full tablet report...
I20240503 14:06:18.352978 18920 heartbeater.cc:503] Master 127.13.158.126:42091 was elected leader, sending a full tablet report...
I20240503 14:06:18.382948 13945 test_util.cc:268] Using random seed: 960291405
I20240503 14:06:18.406145 18643 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60740:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:06:18.408818 18643 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:06:18.468242 18885 tablet_service.cc:1467] Processing CreateTablet for tablet 99d36677e8ca4144b57786d9bd693fe0 (DEFAULT_TABLE table=test-workload [id=07b73d86b8a64021a407625db39c3d5e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 14:06:18.469872 18885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d36677e8ca4144b57786d9bd693fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:18.470109 18883 tablet_service.cc:1467] Processing CreateTablet for tablet 26577847582a4a19804ec9bab95ad978 (DEFAULT_TABLE table=test-workload [id=07b73d86b8a64021a407625db39c3d5e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 14:06:18.468367 18884 tablet_service.cc:1467] Processing CreateTablet for tablet 45c46a6ace434ca3949b816130850824 (DEFAULT_TABLE table=test-workload [id=07b73d86b8a64021a407625db39c3d5e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 14:06:18.472090 18882 tablet_service.cc:1467] Processing CreateTablet for tablet e2816c401ee24977962c4ebca54ff64f (DEFAULT_TABLE table=test-workload [id=07b73d86b8a64021a407625db39c3d5e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 14:06:18.472970 18884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c46a6ace434ca3949b816130850824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:18.473639 18882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2816c401ee24977962c4ebca54ff64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:18.474331 18883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26577847582a4a19804ec9bab95ad978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:18.484745 18810 tablet_service.cc:1467] Processing CreateTablet for tablet 99d36677e8ca4144b57786d9bd693fe0 (DEFAULT_TABLE table=test-workload [id=07b73d86b8a64021a407625db39c3d5e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 14:06:18.484982 18809 tablet_service.cc:1467] Processing CreateTablet for tablet 45c46a6ace434ca3949b816130850824 (DEFAULT_TABLE table=test-workload [id=07b73d86b8a64021a407625db39c3d5e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 14:06:18.486305 18810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d36677e8ca4144b57786d9bd693fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:18.487030 18809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c46a6ace434ca3949b816130850824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:18.489914 18808 tablet_service.cc:1467] Processing CreateTablet for tablet 26577847582a4a19804ec9bab95ad978 (DEFAULT_TABLE table=test-workload [id=07b73d86b8a64021a407625db39c3d5e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 14:06:18.491328 18808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26577847582a4a19804ec9bab95ad978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:18.494323 18807 tablet_service.cc:1467] Processing CreateTablet for tablet e2816c401ee24977962c4ebca54ff64f (DEFAULT_TABLE table=test-workload [id=07b73d86b8a64021a407625db39c3d5e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 14:06:18.495790 18807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2816c401ee24977962c4ebca54ff64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:18.497102 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 45c46a6ace434ca3949b816130850824 (DEFAULT_TABLE table=test-workload [id=07b73d86b8a64021a407625db39c3d5e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 14:06:18.497844 18735 tablet_service.cc:1467] Processing CreateTablet for tablet 99d36677e8ca4144b57786d9bd693fe0 (DEFAULT_TABLE table=test-workload [id=07b73d86b8a64021a407625db39c3d5e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 14:06:18.499615 18733 tablet_service.cc:1467] Processing CreateTablet for tablet 26577847582a4a19804ec9bab95ad978 (DEFAULT_TABLE table=test-workload [id=07b73d86b8a64021a407625db39c3d5e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 14:06:18.501886 18732 tablet_service.cc:1467] Processing CreateTablet for tablet e2816c401ee24977962c4ebca54ff64f (DEFAULT_TABLE table=test-workload [id=07b73d86b8a64021a407625db39c3d5e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 14:06:18.502619 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c46a6ace434ca3949b816130850824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:18.503417 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2816c401ee24977962c4ebca54ff64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:18.504354 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d36677e8ca4144b57786d9bd693fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:18.506028 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26577847582a4a19804ec9bab95ad978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:18.533074 18941 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap starting.
I20240503 14:06:18.539247 18940 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap starting.
I20240503 14:06:18.545532 18941 tablet_bootstrap.cc:654] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:18.545830 18940 tablet_bootstrap.cc:654] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:18.549515 18942 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd: Bootstrap starting.
I20240503 14:06:18.555728 18940 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: No bootstrap required, opened a new log
I20240503 14:06:18.555981 18942 tablet_bootstrap.cc:654] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:18.556120 18941 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: No bootstrap required, opened a new log
I20240503 14:06:18.556315 18940 ts_tablet_manager.cc:1367] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20240503 14:06:18.556548 18941 ts_tablet_manager.cc:1367] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.000s
I20240503 14:06:18.559885 18941 raft_consensus.cc:348] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:18.560168 18940 raft_consensus.cc:348] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:18.560917 18941 raft_consensus.cc:374] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:18.561005 18940 raft_consensus.cc:374] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:18.561259 18941 raft_consensus.cc:729] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92dddf617ffe4b4cb9dfb1f5e8a8e29a, State: Initialized, Role: FOLLOWER
I20240503 14:06:18.561401 18940 raft_consensus.cc:729] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Initialized, Role: FOLLOWER
I20240503 14:06:18.562218 18942 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd: No bootstrap required, opened a new log
I20240503 14:06:18.562125 18941 consensus_queue.cc:260] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:18.562759 18942 ts_tablet_manager.cc:1367] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20240503 14:06:18.562392 18940 consensus_queue.cc:260] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:18.564988 18941 ts_tablet_manager.cc:1398] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240503 14:06:18.565855 18941 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap starting.
I20240503 14:06:18.565732 18942 raft_consensus.cc:348] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:18.566643 18942 raft_consensus.cc:374] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:18.566955 18942 raft_consensus.cc:729] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af02be205214faaaa7f719ee14400bd, State: Initialized, Role: FOLLOWER
I20240503 14:06:18.567695 18942 consensus_queue.cc:260] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:18.572223 18941 tablet_bootstrap.cc:654] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:18.573750 18940 ts_tablet_manager.cc:1398] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: Time spent starting tablet: real 0.017s user 0.009s sys 0.004s
I20240503 14:06:18.580128 18941 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: No bootstrap required, opened a new log
I20240503 14:06:18.580564 18941 ts_tablet_manager.cc:1367] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20240503 14:06:18.582646 18941 raft_consensus.cc:348] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.583382 18941 raft_consensus.cc:374] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:18.583654 18941 raft_consensus.cc:729] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92dddf617ffe4b4cb9dfb1f5e8a8e29a, State: Initialized, Role: FOLLOWER
I20240503 14:06:18.584465 18941 consensus_queue.cc:260] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.585613 18940 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap starting.
I20240503 14:06:18.588989 18941 ts_tablet_manager.cc:1398] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 14:06:18.589982 18941 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap starting.
I20240503 14:06:18.592257 18940 tablet_bootstrap.cc:654] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:18.594028 18942 ts_tablet_manager.cc:1398] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd: Time spent starting tablet: real 0.031s user 0.012s sys 0.019s
I20240503 14:06:18.595168 18942 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd: Bootstrap starting.
I20240503 14:06:18.595690 18941 tablet_bootstrap.cc:654] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:18.600867 18942 tablet_bootstrap.cc:654] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:18.615785 18942 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd: No bootstrap required, opened a new log
I20240503 14:06:18.616236 18942 ts_tablet_manager.cc:1367] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.003s
I20240503 14:06:18.616946 18940 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79: No bootstrap required, opened a new log
I20240503 14:06:18.617064 18941 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: No bootstrap required, opened a new log
I20240503 14:06:18.617488 18940 ts_tablet_manager.cc:1367] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.009s
I20240503 14:06:18.617528 18941 ts_tablet_manager.cc:1367] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.012s
I20240503 14:06:18.619509 18946 raft_consensus.cc:482] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:18.619252 18942 raft_consensus.cc:348] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.620051 18942 raft_consensus.cc:374] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:18.619948 18946 raft_consensus.cc:504] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.620400 18942 raft_consensus.cc:729] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af02be205214faaaa7f719ee14400bd, State: Initialized, Role: FOLLOWER
I20240503 14:06:18.620350 18941 raft_consensus.cc:348] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.621331 18941 raft_consensus.cc:374] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:18.621677 18941 raft_consensus.cc:729] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92dddf617ffe4b4cb9dfb1f5e8a8e29a, State: Initialized, Role: FOLLOWER
I20240503 14:06:18.621469 18942 consensus_queue.cc:260] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.621754 18940 raft_consensus.cc:348] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.622761 18940 raft_consensus.cc:374] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:18.622561 18941 consensus_queue.cc:260] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.623107 18940 raft_consensus.cc:729] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Initialized, Role: FOLLOWER
I20240503 14:06:18.624075 18940 consensus_queue.cc:260] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.626199 18942 ts_tablet_manager.cc:1398] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240503 14:06:18.626904 18940 ts_tablet_manager.cc:1398] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240503 14:06:18.627220 18942 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: Bootstrap starting.
I20240503 14:06:18.628099 18940 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap starting.
I20240503 14:06:18.633383 18942 tablet_bootstrap.cc:654] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:18.634972 18940 tablet_bootstrap.cc:654] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:18.647589 18940 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: No bootstrap required, opened a new log
I20240503 14:06:18.648295 18940 ts_tablet_manager.cc:1367] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.005s
I20240503 14:06:18.651351 18940 raft_consensus.cc:348] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.652290 18940 raft_consensus.cc:374] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:18.652662 18940 raft_consensus.cc:729] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Initialized, Role: FOLLOWER
I20240503 14:06:18.653609 18940 consensus_queue.cc:260] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.659772 18940 ts_tablet_manager.cc:1398] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240503 14:06:18.661032 18940 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap starting.
I20240503 14:06:18.625167 18946 leader_election.cc:288] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
I20240503 14:06:18.667630 18940 tablet_bootstrap.cc:654] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:18.673360 18947 raft_consensus.cc:482] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:18.674036 18947 raft_consensus.cc:504] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:18.678737 18947 leader_election.cc:288] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:18.697696 18940 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: No bootstrap required, opened a new log
I20240503 14:06:18.698335 18940 ts_tablet_manager.cc:1367] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.005s
I20240503 14:06:18.700999 18940 raft_consensus.cc:348] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.703675 18940 raft_consensus.cc:374] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:18.705741 18895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c46a6ace434ca3949b816130850824" candidate_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64118e159acd4c9b89b5267d5d34ee79" is_pre_election: true
I20240503 14:06:18.706738 18895 raft_consensus.cc:2454] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a in term 0.
I20240503 14:06:18.708308 18786 leader_election.cc:302] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64118e159acd4c9b89b5267d5d34ee79, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters:
I20240503 14:06:18.709203 18946 raft_consensus.cc:2781] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:06:18.709558 18946 raft_consensus.cc:482] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:18.709870 18946 raft_consensus.cc:3037] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:18.713625 18942 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: No bootstrap required, opened a new log
I20240503 14:06:18.714165 18942 ts_tablet_manager.cc:1367] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: Time spent bootstrapping tablet: real 0.087s user 0.031s sys 0.053s
I20240503 14:06:18.714213 18941 ts_tablet_manager.cc:1398] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent starting tablet: real 0.096s user 0.031s sys 0.037s
I20240503 14:06:18.717036 18942 raft_consensus.cc:348] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.717803 18942 raft_consensus.cc:374] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:18.717446 18946 raft_consensus.cc:504] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.717993 18941 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap starting.
I20240503 14:06:18.705778 18940 raft_consensus.cc:729] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Initialized, Role: FOLLOWER
I20240503 14:06:18.719800 18946 leader_election.cc:288] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [CANDIDATE]: Term 1 election: Requested vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
I20240503 14:06:18.720286 18942 raft_consensus.cc:729] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af02be205214faaaa7f719ee14400bd, State: Initialized, Role: FOLLOWER
I20240503 14:06:18.720902 18895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c46a6ace434ca3949b816130850824" candidate_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64118e159acd4c9b89b5267d5d34ee79"
I20240503 14:06:18.721541 18895 raft_consensus.cc:3037] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:18.721181 18942 consensus_queue.cc:260] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.723840 18942 ts_tablet_manager.cc:1398] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20240503 14:06:18.724787 18942 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: Bootstrap starting.
I20240503 14:06:18.724866 18941 tablet_bootstrap.cc:654] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:18.729395 18895 raft_consensus.cc:2454] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a in term 1.
I20240503 14:06:18.730820 18942 tablet_bootstrap.cc:654] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: Neither blocks nor log segments found. Creating new log.
I20240503 14:06:18.730749 18786 leader_election.cc:302] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64118e159acd4c9b89b5267d5d34ee79, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters:
I20240503 14:06:18.719272 18940 consensus_queue.cc:260] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.736260 18946 raft_consensus.cc:2781] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:06:18.736819 18946 raft_consensus.cc:686] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 LEADER]: Becoming Leader. State: Replica: 92dddf617ffe4b4cb9dfb1f5e8a8e29a, State: Running, Role: LEADER
I20240503 14:06:18.737715 18946 consensus_queue.cc:237] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.744699 18948 raft_consensus.cc:482] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:18.745450 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c46a6ace434ca3949b816130850824" candidate_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd" is_pre_election: true
I20240503 14:06:18.745249 18948 raft_consensus.cc:504] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:18.745978 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c46a6ace434ca3949b816130850824" candidate_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd"
I20240503 14:06:18.746201 18745 raft_consensus.cc:2454] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a in term 0.
I20240503 14:06:18.745433 18940 ts_tablet_manager.cc:1398] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: Time spent starting tablet: real 0.047s user 0.002s sys 0.004s
I20240503 14:06:18.750850 18948 leader_election.cc:288] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:18.753543 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd" is_pre_election: true
I20240503 14:06:18.754252 18744 raft_consensus.cc:2454] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 0.
I20240503 14:06:18.755705 18859 leader_election.cc:302] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 64118e159acd4c9b89b5267d5d34ee79; no voters:
I20240503 14:06:18.756579 18947 raft_consensus.cc:2781] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:06:18.756930 18947 raft_consensus.cc:482] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:18.757279 18947 raft_consensus.cc:3037] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:18.757709 18942 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: No bootstrap required, opened a new log
I20240503 14:06:18.758213 18942 ts_tablet_manager.cc:1367] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.003s
I20240503 14:06:18.759618 18941 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: No bootstrap required, opened a new log
I20240503 14:06:18.773703 18820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" is_pre_election: true
I20240503 14:06:18.764786 18947 raft_consensus.cc:504] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:18.761255 18942 raft_consensus.cc:348] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.765096 18641 catalog_manager.cc:5484] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a reported cstate change: term changed from 0 to 1, leader changed from <none> to 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66). New cstate: current_term: 1 leader_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } health_report { overall_health: UNKNOWN } } }
I20240503 14:06:18.775142 18942 raft_consensus.cc:374] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:18.775135 18820 raft_consensus.cc:2454] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 0.
I20240503 14:06:18.775525 18942 raft_consensus.cc:729] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af02be205214faaaa7f719ee14400bd, State: Initialized, Role: FOLLOWER
I20240503 14:06:18.776350 18942 consensus_queue.cc:260] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.777105 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd"
I20240503 14:06:18.777761 18744 raft_consensus.cc:3037] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:18.778699 18942 ts_tablet_manager.cc:1398] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: Time spent starting tablet: real 0.020s user 0.008s sys 0.012s
I20240503 14:06:18.782925 18947 leader_election.cc:288] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 1 election: Requested vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:18.776989 18941 ts_tablet_manager.cc:1367] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent bootstrapping tablet: real 0.059s user 0.019s sys 0.008s
I20240503 14:06:18.784731 18744 raft_consensus.cc:2454] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 1.
I20240503 14:06:18.785811 18859 leader_election.cc:302] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 64118e159acd4c9b89b5267d5d34ee79; no voters:
I20240503 14:06:18.786072 18941 raft_consensus.cc:348] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.786813 18941 raft_consensus.cc:374] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:18.786645 18947 raft_consensus.cc:2781] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:06:18.787189 18941 raft_consensus.cc:729] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92dddf617ffe4b4cb9dfb1f5e8a8e29a, State: Initialized, Role: FOLLOWER
I20240503 14:06:18.788019 18941 consensus_queue.cc:260] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.790352 18941 ts_tablet_manager.cc:1398] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:06:18.791479 18895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "0af02be205214faaaa7f719ee14400bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64118e159acd4c9b89b5267d5d34ee79" is_pre_election: true
I20240503 14:06:18.792385 18820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a"
I20240503 14:06:18.792964 18820 raft_consensus.cc:3037] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:18.795629 18947 raft_consensus.cc:686] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 LEADER]: Becoming Leader. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Running, Role: LEADER
I20240503 14:06:18.796496 18947 consensus_queue.cc:237] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:18.803114 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "0af02be205214faaaa7f719ee14400bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" is_pre_election: true
I20240503 14:06:18.803735 18820 raft_consensus.cc:2454] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 1.
I20240503 14:06:18.804402 18711 leader_election.cc:302] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd; no voters: 64118e159acd4c9b89b5267d5d34ee79, 92dddf617ffe4b4cb9dfb1f5e8a8e29a
I20240503 14:06:18.805274 18948 raft_consensus.cc:2726] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:06:18.808317 18641 catalog_manager.cc:5484] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67). New cstate: current_term: 1 leader_uuid: "64118e159acd4c9b89b5267d5d34ee79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } health_report { overall_health: UNKNOWN } } }
I20240503 14:06:18.881646 18947 raft_consensus.cc:482] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:18.882086 18947 raft_consensus.cc:504] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.883594 18947 leader_election.cc:288] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:18.884455 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577847582a4a19804ec9bab95ad978" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd" is_pre_election: true
I20240503 14:06:18.884626 18820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577847582a4a19804ec9bab95ad978" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" is_pre_election: true
I20240503 14:06:18.885003 18744 raft_consensus.cc:2454] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 0.
I20240503 14:06:18.885262 18820 raft_consensus.cc:2454] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 0.
I20240503 14:06:18.886058 18859 leader_election.cc:302] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 64118e159acd4c9b89b5267d5d34ee79; no voters:
I20240503 14:06:18.886739 18947 raft_consensus.cc:2781] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:06:18.886999 18947 raft_consensus.cc:482] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:18.887231 18947 raft_consensus.cc:3037] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:18.891745 18947 raft_consensus.cc:504] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.893224 18947 leader_election.cc:288] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 1 election: Requested vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:18.894021 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577847582a4a19804ec9bab95ad978" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd"
I20240503 14:06:18.894201 18820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577847582a4a19804ec9bab95ad978" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a"
I20240503 14:06:18.894704 18744 raft_consensus.cc:3037] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:18.894757 18820 raft_consensus.cc:3037] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:18.901785 18820 raft_consensus.cc:2454] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 1.
I20240503 14:06:18.901785 18744 raft_consensus.cc:2454] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 1.
I20240503 14:06:18.903247 18861 leader_election.cc:302] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64118e159acd4c9b89b5267d5d34ee79, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters:
I20240503 14:06:18.903972 18947 raft_consensus.cc:2781] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:06:18.904454 18947 raft_consensus.cc:686] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 LEADER]: Becoming Leader. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Running, Role: LEADER
I20240503 14:06:18.905138 18947 consensus_queue.cc:237] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
W20240503 14:06:18.906508 18691 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:06:18.911412 18641 catalog_manager.cc:5484] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67). New cstate: current_term: 1 leader_uuid: "64118e159acd4c9b89b5267d5d34ee79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } health_report { overall_health: HEALTHY } } }
I20240503 14:06:18.934468 18947 raft_consensus.cc:482] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:18.934927 18947 raft_consensus.cc:504] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.936499 18947 leader_election.cc:288] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:18.937278 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2816c401ee24977962c4ebca54ff64f" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd" is_pre_election: true
I20240503 14:06:18.937469 18820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2816c401ee24977962c4ebca54ff64f" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" is_pre_election: true
I20240503 14:06:18.937834 18744 raft_consensus.cc:2454] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 0.
I20240503 14:06:18.938038 18820 raft_consensus.cc:2454] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 0.
I20240503 14:06:18.938752 18859 leader_election.cc:302] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 64118e159acd4c9b89b5267d5d34ee79; no voters:
I20240503 14:06:18.939429 18947 raft_consensus.cc:2781] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:06:18.939767 18947 raft_consensus.cc:482] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:18.940016 18947 raft_consensus.cc:3037] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:18.944806 18947 raft_consensus.cc:504] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.946323 18947 leader_election.cc:288] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 1 election: Requested vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:18.947170 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2816c401ee24977962c4ebca54ff64f" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd"
I20240503 14:06:18.947357 18820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2816c401ee24977962c4ebca54ff64f" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a"
I20240503 14:06:18.947712 18744 raft_consensus.cc:3037] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:18.947883 18820 raft_consensus.cc:3037] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:18.952483 18820 raft_consensus.cc:2454] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 1.
I20240503 14:06:18.952478 18744 raft_consensus.cc:2454] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 1.
I20240503 14:06:18.953675 18859 leader_election.cc:302] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 64118e159acd4c9b89b5267d5d34ee79, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters:
I20240503 14:06:18.954363 18947 raft_consensus.cc:2781] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:06:18.954823 18947 raft_consensus.cc:686] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 1 LEADER]: Becoming Leader. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Running, Role: LEADER
I20240503 14:06:18.955389 18947 consensus_queue.cc:237] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:18.961988 18641 catalog_manager.cc:5484] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67). New cstate: current_term: 1 leader_uuid: "64118e159acd4c9b89b5267d5d34ee79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } health_report { overall_health: HEALTHY } } }
I20240503 14:06:19.038524 13945 tablet_server.cc:178] TabletServer@127.13.158.65:0 shutting down...
I20240503 14:06:19.059877 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:06:19.060858 13945 tablet_replica.cc:330] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd: stopping tablet replica
I20240503 14:06:19.061448 13945 raft_consensus.cc:2229] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:19.061843 13945 raft_consensus.cc:2258] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:19.063721 13945 tablet_replica.cc:330] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: stopping tablet replica
I20240503 14:06:19.064251 13945 raft_consensus.cc:2229] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:19.064594 13945 raft_consensus.cc:2258] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:19.066303 13945 tablet_replica.cc:330] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: stopping tablet replica
I20240503 14:06:19.066852 13945 raft_consensus.cc:2229] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:19.067194 13945 raft_consensus.cc:2258] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:19.068872 13945 tablet_replica.cc:330] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd: stopping tablet replica
I20240503 14:06:19.069321 13945 raft_consensus.cc:2229] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:19.069653 13945 raft_consensus.cc:2258] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:19.091516 13945 tablet_server.cc:194] TabletServer@127.13.158.65:0 shutdown complete.
I20240503 14:06:19.103427 13945 tablet_server.cc:178] TabletServer@127.13.158.66:0 shutting down...
I20240503 14:06:19.149228 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:06:19.150061 13945 tablet_replica.cc:330] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: stopping tablet replica
I20240503 14:06:19.150842 13945 raft_consensus.cc:2229] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:06:19.151427 13945 pending_rounds.cc:62] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Trying to abort 1 pending ops.
I20240503 14:06:19.151611 13945 pending_rounds.cc:68] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023596252119785472 op_type: NO_OP noop_request { }
I20240503 14:06:19.151926 13945 raft_consensus.cc:2866] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:06:19.152148 13945 raft_consensus.cc:2258] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:19.153990 13945 tablet_replica.cc:330] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: stopping tablet replica
I20240503 14:06:19.154548 13945 raft_consensus.cc:2229] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:19.154870 13945 raft_consensus.cc:2258] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:19.156621 13945 tablet_replica.cc:330] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: stopping tablet replica
I20240503 14:06:19.157063 13945 raft_consensus.cc:2229] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:19.157361 13945 raft_consensus.cc:2258] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:19.158857 13945 tablet_replica.cc:330] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: stopping tablet replica
I20240503 14:06:19.159255 13945 raft_consensus.cc:2229] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:19.159546 13945 raft_consensus.cc:2258] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:19.181218 13945 tablet_server.cc:194] TabletServer@127.13.158.66:0 shutdown complete.
I20240503 14:06:19.193136 13945 tablet_server.cc:178] TabletServer@127.13.158.67:0 shutting down...
W20240503 14:06:19.207183 18859 consensus_peers.cc:485] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 -> Peer 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721): Couldn't send request to peer 0af02be205214faaaa7f719ee14400bd. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:40721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:19.213137 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:06:19.213900 13945 tablet_replica.cc:330] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79: stopping tablet replica
I20240503 14:06:19.214461 13945 raft_consensus.cc:2229] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:19.214861 13945 raft_consensus.cc:2258] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:19.216461 13945 tablet_replica.cc:330] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: stopping tablet replica
I20240503 14:06:19.216950 13945 raft_consensus.cc:2229] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:06:19.217490 13945 pending_rounds.cc:62] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: Trying to abort 1 pending ops.
I20240503 14:06:19.217643 13945 pending_rounds.cc:68] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023596252802744320 op_type: NO_OP noop_request { }
I20240503 14:06:19.217921 13945 raft_consensus.cc:2866] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:06:19.218165 13945 raft_consensus.cc:2258] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:19.220144 13945 tablet_replica.cc:330] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: stopping tablet replica
I20240503 14:06:19.220629 13945 raft_consensus.cc:2229] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:06:19.221160 13945 pending_rounds.cc:62] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: Trying to abort 1 pending ops.
I20240503 14:06:19.221309 13945 pending_rounds.cc:68] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023596253007609856 op_type: NO_OP noop_request { }
I20240503 14:06:19.221580 13945 raft_consensus.cc:2866] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:06:19.221832 13945 raft_consensus.cc:2258] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:19.223745 13945 tablet_replica.cc:330] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: stopping tablet replica
I20240503 14:06:19.224195 13945 raft_consensus.cc:2229] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:06:19.224799 13945 pending_rounds.cc:62] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: Trying to abort 1 pending ops.
I20240503 14:06:19.224965 13945 pending_rounds.cc:68] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023596252359991296 op_type: NO_OP noop_request { }
I20240503 14:06:19.225225 13945 raft_consensus.cc:2866] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:06:19.225466 13945 raft_consensus.cc:2258] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:19.247910 13945 tablet_server.cc:194] TabletServer@127.13.158.67:0 shutdown complete.
I20240503 14:06:19.259649 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:06:19.265940 18972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:19.266891 18973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:19.267515 18975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:19.268081 13945 server_base.cc:1034] running on GCE node
I20240503 14:06:19.269171 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:19.269461 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:19.269699 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745179269686 us; error 0 us; skew 500 ppm
I20240503 14:06:19.270282 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:19.278374 13945 webserver.cc:460] Webserver started at http://127.13.158.68:45621/ using document root <none> and password file <none>
I20240503 14:06:19.279007 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:19.279287 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:06:19.279615 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:06:19.280804 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-3-root/instance:
uuid: "5737a8388fc14c58a4fb5a3d205d083e"
format_stamp: "Formatted at 2024-05-03 14:06:19 on dist-test-slave-jm3r"
I20240503 14:06:19.285424 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:06:19.288825 18980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:19.289687 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:06:19.290073 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-3-root
uuid: "5737a8388fc14c58a4fb5a3d205d083e"
format_stamp: "Formatted at 2024-05-03 14:06:19 on dist-test-slave-jm3r"
I20240503 14:06:19.290503 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:19.313761 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:19.314891 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:19.316398 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:06:19.318728 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:06:19.319027 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:19.319327 13945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:06:19.319569 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:19.356307 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.68:41469
I20240503 14:06:19.356402 19042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.68:41469 every 8 connection(s)
I20240503 14:06:19.369812 19043 heartbeater.cc:347] Connected to a master server at 127.13.158.126:42091
I20240503 14:06:19.370252 19043 heartbeater.cc:464] Registering TS with master...
I20240503 14:06:19.370970 19043 heartbeater.cc:511] Master 127.13.158.126:42091 requested a full tablet report, sending...
I20240503 14:06:19.372859 18641 ts_manager.cc:194] Registered new tserver with Master: 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68:41469)
I20240503 14:06:19.374310 18641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42522
W20240503 14:06:19.917971 18691 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.13.158.66:38261: connect: Connection refused (error 111)
W20240503 14:06:19.924074 18691 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.13.158.66:38261: connect: Connection refused (error 111)
W20240503 14:06:19.924500 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.13.158.66:38261: connect: Connection refused (error 111)
W20240503 14:06:19.928823 18691 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:19.929211 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:19.933327 18691 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:19.933696 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
I20240503 14:06:20.377048 19043 heartbeater.cc:503] Master 127.13.158.126:42091 was elected leader, sending a full tablet report...
W20240503 14:06:20.965523 18691 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:20.969223 18630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240503 14:06:20.971918 18691 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:20.972148 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:20.976286 18691 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.13.158.66:38261: connect: Connection refused (error 111)
W20240503 14:06:20.976550 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.13.158.66:38261: connect: Connection refused (error 111)
W20240503 14:06:20.980455 18691 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:20.980695 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:21.992833 18691 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:21.998706 18691 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:21.998966 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:22.003414 18691 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.13.158.66:38261: connect: Connection refused (error 111)
W20240503 14:06:22.003666 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.13.158.66:38261: connect: Connection refused (error 111)
W20240503 14:06:22.008123 18691 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:22.008387 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:23.021494 18691 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:23.031040 18691 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:23.031298 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:23.036931 18691 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.13.158.66:38261: connect: Connection refused (error 111)
W20240503 14:06:23.037300 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.13.158.66:38261: connect: Connection refused (error 111)
W20240503 14:06:23.042464 18691 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:23.042737 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.13.158.67:37481: connect: Connection refused (error 111)
W20240503 14:06:24.044811 18691 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 64118e159acd4c9b89b5267d5d34ee79 not available for placement
W20240503 14:06:25.046779 18691 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 64118e159acd4c9b89b5267d5d34ee79 not available for placement
I20240503 14:06:25.395699 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:06:25.404600 19059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:25.405937 19060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:25.407949 19062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:25.411515 13945 server_base.cc:1034] running on GCE node
I20240503 14:06:25.412635 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:25.412864 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:25.413033 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745185413017 us; error 0 us; skew 500 ppm
I20240503 14:06:25.413631 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:25.422104 13945 webserver.cc:460] Webserver started at http://127.13.158.65:44957/ using document root <none> and password file <none>
I20240503 14:06:25.422716 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:25.422896 13945 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 14:06:25.426651 13945 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:06:25.429759 19067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:25.430588 13945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:06:25.430896 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-0-root
uuid: "0af02be205214faaaa7f719ee14400bd"
format_stamp: "Formatted at 2024-05-03 14:06:16 on dist-test-slave-jm3r"
I20240503 14:06:25.431191 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:25.444509 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:25.445716 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:25.447176 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:06:25.451073 19074 ts_tablet_manager.cc:522] Loading tablet metadata (0/4 complete)
I20240503 14:06:25.474007 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240503 14:06:25.474306 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.024s user 0.000s sys 0.002s
I20240503 14:06:25.474567 13945 ts_tablet_manager.cc:574] Registering tablets (0/4 complete)
I20240503 14:06:25.479431 19074 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: Bootstrap starting.
I20240503 14:06:25.494167 19074 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:25.495258 19074 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: Bootstrap complete.
I20240503 14:06:25.496111 19074 ts_tablet_manager.cc:1367] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.002s
I20240503 14:06:25.499186 19074 raft_consensus.cc:348] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.499879 13945 ts_tablet_manager.cc:590] Registered 4 tablets
I20240503 14:06:25.499989 19074 raft_consensus.cc:729] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af02be205214faaaa7f719ee14400bd, State: Initialized, Role: FOLLOWER
I20240503 14:06:25.500229 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.026s user 0.024s sys 0.000s
I20240503 14:06:25.501155 19074 consensus_queue.cc:260] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.504603 19074 ts_tablet_manager.cc:1398] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: Time spent starting tablet: real 0.008s user 0.007s sys 0.002s
I20240503 14:06:25.505569 19074 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd: Bootstrap starting.
I20240503 14:06:25.521036 19074 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:25.522012 19074 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd: Bootstrap complete.
I20240503 14:06:25.522876 19074 ts_tablet_manager.cc:1367] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240503 14:06:25.525543 19074 raft_consensus.cc:348] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.526378 19074 raft_consensus.cc:374] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:06:25.526715 19074 raft_consensus.cc:729] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af02be205214faaaa7f719ee14400bd, State: Initialized, Role: FOLLOWER
I20240503 14:06:25.527488 19074 consensus_queue.cc:260] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.529861 19074 ts_tablet_manager.cc:1398] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:06:25.530896 19074 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: Bootstrap starting.
I20240503 14:06:25.545612 19074 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:25.546361 19074 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: Bootstrap complete.
I20240503 14:06:25.547032 19074 ts_tablet_manager.cc:1367] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20240503 14:06:25.549095 19074 raft_consensus.cc:348] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.549683 19074 raft_consensus.cc:729] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af02be205214faaaa7f719ee14400bd, State: Initialized, Role: FOLLOWER
I20240503 14:06:25.550333 19074 consensus_queue.cc:260] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.552122 19074 ts_tablet_manager.cc:1398] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240503 14:06:25.552845 19074 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd: Bootstrap starting.
I20240503 14:06:25.556561 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.65:40721
I20240503 14:06:25.556679 19134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.65:40721 every 8 connection(s)
W20240503 14:06:25.561966 19136 tablet.cc:2369] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:06:25.562620 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:06:25.573732 19074 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:25.575063 19074 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd: Bootstrap complete.
I20240503 14:06:25.576016 19074 ts_tablet_manager.cc:1367] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.012s
I20240503 14:06:25.579324 19074 raft_consensus.cc:348] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:25.580711 19074 raft_consensus.cc:729] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af02be205214faaaa7f719ee14400bd, State: Initialized, Role: FOLLOWER
W20240503 14:06:25.582347 19140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:06:25.582989 19141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:25.583159 19074 consensus_queue.cc:260] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:25.585525 19074 ts_tablet_manager.cc:1398] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
W20240503 14:06:25.586392 19143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:25.587044 13945 server_base.cc:1034] running on GCE node
I20240503 14:06:25.587359 19135 heartbeater.cc:347] Connected to a master server at 127.13.158.126:42091
I20240503 14:06:25.587778 19135 heartbeater.cc:464] Registering TS with master...
I20240503 14:06:25.588183 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:25.588470 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:25.588677 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745185588656 us; error 0 us; skew 500 ppm
I20240503 14:06:25.588738 19135 heartbeater.cc:511] Master 127.13.158.126:42091 requested a full tablet report, sending...
I20240503 14:06:25.589489 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:25.592366 13945 webserver.cc:460] Webserver started at http://127.13.158.66:39891/ using document root <none> and password file <none>
I20240503 14:06:25.592881 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:25.593042 13945 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 14:06:25.593392 18641 ts_manager.cc:194] Re-registered known tserver with Master: 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721)
I20240503 14:06:25.596947 13945 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:06:25.598213 18641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42528
I20240503 14:06:25.600390 19148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:25.601038 19135 heartbeater.cc:503] Master 127.13.158.126:42091 was elected leader, sending a full tablet report...
I20240503 14:06:25.601305 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:06:25.601682 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-1-root
uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a"
format_stamp: "Formatted at 2024-05-03 14:06:17 on dist-test-slave-jm3r"
I20240503 14:06:25.602088 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:25.622400 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:25.623607 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:25.625036 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:06:25.650763 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240503 14:06:25.651082 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.023s user 0.002s sys 0.000s
I20240503 14:06:25.656124 19155 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap starting.
I20240503 14:06:25.666744 19155 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:25.667424 19155 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap complete.
I20240503 14:06:25.667989 19155 ts_tablet_manager.cc:1367] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 14:06:25.668289 13945 ts_tablet_manager.cc:590] Registered 4 tablets
I20240503 14:06:25.668540 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.017s user 0.013s sys 0.004s
I20240503 14:06:25.670159 19155 raft_consensus.cc:348] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.670827 19155 raft_consensus.cc:729] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92dddf617ffe4b4cb9dfb1f5e8a8e29a, State: Initialized, Role: FOLLOWER
I20240503 14:06:25.671476 19155 consensus_queue.cc:260] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.673974 19155 ts_tablet_manager.cc:1398] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 14:06:25.674726 19155 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap starting.
I20240503 14:06:25.688544 19155 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:06:25.689884 19155 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap complete.
I20240503 14:06:25.690536 19155 ts_tablet_manager.cc:1367] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20240503 14:06:25.692955 19155 raft_consensus.cc:348] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.693796 19155 raft_consensus.cc:729] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92dddf617ffe4b4cb9dfb1f5e8a8e29a, State: Initialized, Role: FOLLOWER
I20240503 14:06:25.694746 19155 consensus_queue.cc:260] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.696924 19155 ts_tablet_manager.cc:1398] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:06:25.698108 19155 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap starting.
I20240503 14:06:25.711310 19155 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:25.712244 19155 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap complete.
I20240503 14:06:25.713076 19155 ts_tablet_manager.cc:1367] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20240503 14:06:25.715873 19155 raft_consensus.cc:348] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.716714 19155 raft_consensus.cc:729] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92dddf617ffe4b4cb9dfb1f5e8a8e29a, State: Initialized, Role: FOLLOWER
I20240503 14:06:25.717552 19155 consensus_queue.cc:260] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.719857 19155 ts_tablet_manager.cc:1398] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240503 14:06:25.720834 19155 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap starting.
I20240503 14:06:25.722851 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.66:38261
I20240503 14:06:25.722955 19215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.66:38261 every 8 connection(s)
I20240503 14:06:25.728113 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:06:25.735810 19155 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:25.737222 19155 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Bootstrap complete.
I20240503 14:06:25.738073 19155 ts_tablet_manager.cc:1367] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
W20240503 14:06:25.737586 19220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:25.740607 19216 heartbeater.cc:347] Connected to a master server at 127.13.158.126:42091
I20240503 14:06:25.741103 19216 heartbeater.cc:464] Registering TS with master...
I20240503 14:06:25.742185 19216 heartbeater.cc:511] Master 127.13.158.126:42091 requested a full tablet report, sending...
I20240503 14:06:25.741118 19155 raft_consensus.cc:348] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:25.742744 19155 raft_consensus.cc:729] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92dddf617ffe4b4cb9dfb1f5e8a8e29a, State: Initialized, Role: FOLLOWER
I20240503 14:06:25.743539 19155 consensus_queue.cc:260] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
W20240503 14:06:25.744859 19221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:25.746459 19155 ts_tablet_manager.cc:1398] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
W20240503 14:06:25.749629 19223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:06:25.750910 13945 server_base.cc:1034] running on GCE node
I20240503 14:06:25.751132 18641 ts_manager.cc:194] Re-registered known tserver with Master: 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:25.752103 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:06:25.752424 13945 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:06:25.752640 13945 hybrid_clock.cc:648] HybridClock initialized: now 1714745185752619 us; error 0 us; skew 500 ppm
I20240503 14:06:25.753468 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:06:25.756914 13945 webserver.cc:460] Webserver started at http://127.13.158.67:39445/ using document root <none> and password file <none>
I20240503 14:06:25.757562 13945 fs_manager.cc:359] Metadata directory not provided
I20240503 14:06:25.757822 13945 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 14:06:25.758330 18641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42544
I20240503 14:06:25.761827 19216 heartbeater.cc:503] Master 127.13.158.126:42091 was elected leader, sending a full tablet report...
I20240503 14:06:25.762852 13945 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:06:25.767205 19228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:06:25.768414 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:06:25.768891 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-2-root
uuid: "64118e159acd4c9b89b5267d5d34ee79"
format_stamp: "Formatted at 2024-05-03 14:06:17 on dist-test-slave-jm3r"
I20240503 14:06:25.769279 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskld9eS5/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714745036514908-13945-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:06:25.783121 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:06:25.784307 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:06:25.785778 13945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:06:25.813844 13945 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240503 14:06:25.814332 13945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.026s user 0.002s sys 0.000s
I20240503 14:06:25.820845 19235 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap starting.
I20240503 14:06:25.833809 19235 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:06:25.834272 13945 ts_tablet_manager.cc:590] Registered 4 tablets
I20240503 14:06:25.834604 13945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.020s user 0.020s sys 0.000s
I20240503 14:06:25.834770 19235 tablet_bootstrap.cc:492] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap complete.
I20240503 14:06:25.835597 19235 ts_tablet_manager.cc:1367] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20240503 14:06:25.838001 19235 raft_consensus.cc:348] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.838821 19235 raft_consensus.cc:729] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Initialized, Role: FOLLOWER
I20240503 14:06:25.839478 19235 consensus_queue.cc:260] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.842465 19235 ts_tablet_manager.cc:1398] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 14:06:25.843254 19235 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap starting.
I20240503 14:06:25.854934 19235 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:25.855839 19235 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap complete.
I20240503 14:06:25.856674 19235 ts_tablet_manager.cc:1367] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.001s
I20240503 14:06:25.859184 19235 raft_consensus.cc:348] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.859824 19235 raft_consensus.cc:729] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Initialized, Role: FOLLOWER
I20240503 14:06:25.860513 19235 consensus_queue.cc:260] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.862673 19235 ts_tablet_manager.cc:1398] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79: Time spent starting tablet: real 0.006s user 0.001s sys 0.007s
I20240503 14:06:25.863441 19235 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap starting.
I20240503 14:06:25.877472 19235 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:06:25.878758 19235 tablet_bootstrap.cc:492] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap complete.
I20240503 14:06:25.879578 19235 ts_tablet_manager.cc:1367] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20240503 14:06:25.882635 19235 raft_consensus.cc:348] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.883790 19235 raft_consensus.cc:729] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Initialized, Role: FOLLOWER
I20240503 14:06:25.884697 19235 consensus_queue.cc:260] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.887142 19235 ts_tablet_manager.cc:1398] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240503 14:06:25.888144 19235 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap starting.
I20240503 14:06:25.888908 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.158.67:37481
I20240503 14:06:25.889005 19295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.158.67:37481 every 8 connection(s)
I20240503 14:06:25.903129 19235 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:06:25.904376 19235 tablet_bootstrap.cc:492] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: Bootstrap complete.
I20240503 14:06:25.905234 19235 ts_tablet_manager.cc:1367] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.012s
I20240503 14:06:25.906147 19296 heartbeater.cc:347] Connected to a master server at 127.13.158.126:42091
I20240503 14:06:25.906875 19296 heartbeater.cc:464] Registering TS with master...
I20240503 14:06:25.908025 19296 heartbeater.cc:511] Master 127.13.158.126:42091 requested a full tablet report, sending...
I20240503 14:06:25.907778 19235 raft_consensus.cc:348] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:25.909158 19235 raft_consensus.cc:729] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Initialized, Role: FOLLOWER
I20240503 14:06:25.909922 19235 consensus_queue.cc:260] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:25.911927 19235 ts_tablet_manager.cc:1398] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240503 14:06:25.914634 18641 ts_manager.cc:194] Re-registered known tserver with Master: 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
I20240503 14:06:25.918999 18641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42556
I20240503 14:06:25.921969 19296 heartbeater.cc:503] Master 127.13.158.126:42091 was elected leader, sending a full tablet report...
I20240503 14:06:25.971659 19077 raft_consensus.cc:482] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:25.972261 19077 raft_consensus.cc:504] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:25.974210 19077 leader_election.cc:288] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261), 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
I20240503 14:06:25.985337 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c46a6ace434ca3949b816130850824" candidate_uuid: "0af02be205214faaaa7f719ee14400bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" is_pre_election: true
I20240503 14:06:25.986598 19190 raft_consensus.cc:2379] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0af02be205214faaaa7f719ee14400bd in current term 1: Already voted for candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a in this term.
I20240503 14:06:25.988534 19270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c46a6ace434ca3949b816130850824" candidate_uuid: "0af02be205214faaaa7f719ee14400bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64118e159acd4c9b89b5267d5d34ee79" is_pre_election: true
I20240503 14:06:25.989360 19270 raft_consensus.cc:2379] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0af02be205214faaaa7f719ee14400bd in current term 1: Already voted for candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a in this term.
I20240503 14:06:25.990489 19071 leader_election.cc:302] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd; no voters: 64118e159acd4c9b89b5267d5d34ee79, 92dddf617ffe4b4cb9dfb1f5e8a8e29a
I20240503 14:06:25.991395 19077 raft_consensus.cc:3037] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:06:25.996783 19077 raft_consensus.cc:2726] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:06:26.069981 18691 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 64118e159acd4c9b89b5267d5d34ee79 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } }
W20240503 14:06:26.074574 18691 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 99d36677e8ca4144b57786d9bd693fe0, TS 0af02be205214faaaa7f719ee14400bd -> TS 5737a8388fc14c58a4fb5a3d205d083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
W20240503 14:06:26.075093 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 99d36677e8ca4144b57786d9bd693fe0, TS 0af02be205214faaaa7f719ee14400bd -> TS 5737a8388fc14c58a4fb5a3d205d083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
W20240503 14:06:26.087466 18691 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 45c46a6ace434ca3949b816130850824, TS 64118e159acd4c9b89b5267d5d34ee79 -> TS 5737a8388fc14c58a4fb5a3d205d083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
W20240503 14:06:26.088179 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 45c46a6ace434ca3949b816130850824, TS 64118e159acd4c9b89b5267d5d34ee79 -> TS 5737a8388fc14c58a4fb5a3d205d083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
W20240503 14:06:26.092386 18691 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e2816c401ee24977962c4ebca54ff64f, TS 92dddf617ffe4b4cb9dfb1f5e8a8e29a -> TS 5737a8388fc14c58a4fb5a3d205d083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
W20240503 14:06:26.092820 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet e2816c401ee24977962c4ebca54ff64f, TS 92dddf617ffe4b4cb9dfb1f5e8a8e29a -> TS 5737a8388fc14c58a4fb5a3d205d083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:26.911906 19304 raft_consensus.cc:482] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:26.912482 19304 raft_consensus.cc:504] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:26.914057 19304 leader_election.cc:288] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261), 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
I20240503 14:06:26.915189 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577847582a4a19804ec9bab95ad978" candidate_uuid: "0af02be205214faaaa7f719ee14400bd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" is_pre_election: true
I20240503 14:06:26.915545 19270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577847582a4a19804ec9bab95ad978" candidate_uuid: "0af02be205214faaaa7f719ee14400bd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64118e159acd4c9b89b5267d5d34ee79" is_pre_election: true
I20240503 14:06:26.915861 19190 raft_consensus.cc:2454] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0af02be205214faaaa7f719ee14400bd in term 1.
I20240503 14:06:26.916363 19270 raft_consensus.cc:2396] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0af02be205214faaaa7f719ee14400bd for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 14:06:26.916822 19071 leader_election.cc:302] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters:
I20240503 14:06:26.917580 19304 raft_consensus.cc:2781] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:06:26.917868 19304 raft_consensus.cc:482] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:26.918133 19304 raft_consensus.cc:3037] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:26.923089 19304 raft_consensus.cc:504] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:26.924681 19304 leader_election.cc:288] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [CANDIDATE]: Term 2 election: Requested vote from peers 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261), 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
I20240503 14:06:26.925737 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577847582a4a19804ec9bab95ad978" candidate_uuid: "0af02be205214faaaa7f719ee14400bd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a"
I20240503 14:06:26.926049 19270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577847582a4a19804ec9bab95ad978" candidate_uuid: "0af02be205214faaaa7f719ee14400bd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64118e159acd4c9b89b5267d5d34ee79"
I20240503 14:06:26.926333 19190 raft_consensus.cc:3037] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:26.926669 19270 raft_consensus.cc:3037] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:26.932735 19190 raft_consensus.cc:2454] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0af02be205214faaaa7f719ee14400bd in term 2.
I20240503 14:06:26.933182 19270 raft_consensus.cc:2396] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0af02be205214faaaa7f719ee14400bd for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 14:06:26.934185 19071 leader_election.cc:302] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters:
I20240503 14:06:26.935124 19304 raft_consensus.cc:2781] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:06:26.936022 19304 raft_consensus.cc:686] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 2 LEADER]: Becoming Leader. State: Replica: 0af02be205214faaaa7f719ee14400bd, State: Running, Role: LEADER
I20240503 14:06:26.936830 19304 consensus_queue.cc:237] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:26.943331 18641 catalog_manager.cc:5484] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd reported cstate change: term changed from 1 to 2, leader changed from 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67) to 0af02be205214faaaa7f719ee14400bd (127.13.158.65). New cstate: current_term: 2 leader_uuid: "0af02be205214faaaa7f719ee14400bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } health_report { overall_health: UNKNOWN } } }
W20240503 14:06:27.111631 18691 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240503 14:06:27.115717 18691 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 26577847582a4a19804ec9bab95ad978, TS 64118e159acd4c9b89b5267d5d34ee79 -> TS 5737a8388fc14c58a4fb5a3d205d083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
W20240503 14:06:27.116001 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 26577847582a4a19804ec9bab95ad978, TS 64118e159acd4c9b89b5267d5d34ee79 -> TS 5737a8388fc14c58a4fb5a3d205d083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
W20240503 14:06:27.119848 18691 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e2816c401ee24977962c4ebca54ff64f, TS 0af02be205214faaaa7f719ee14400bd -> TS 5737a8388fc14c58a4fb5a3d205d083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
W20240503 14:06:27.120121 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet e2816c401ee24977962c4ebca54ff64f, TS 0af02be205214faaaa7f719ee14400bd -> TS 5737a8388fc14c58a4fb5a3d205d083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.123698 19313 raft_consensus.cc:482] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:06:27.123908 18691 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 99d36677e8ca4144b57786d9bd693fe0, TS 92dddf617ffe4b4cb9dfb1f5e8a8e29a -> TS 5737a8388fc14c58a4fb5a3d205d083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
W20240503 14:06:27.124287 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 99d36677e8ca4144b57786d9bd693fe0, TS 92dddf617ffe4b4cb9dfb1f5e8a8e29a -> TS 5737a8388fc14c58a4fb5a3d205d083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:27.124344 19313 raft_consensus.cc:504] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.126822 19313 leader_election.cc:288] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
I20240503 14:06:27.137411 19109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c46a6ace434ca3949b816130850824" candidate_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd" is_pre_election: true
I20240503 14:06:27.138098 19109 raft_consensus.cc:2454] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a in term 1.
I20240503 14:06:27.138963 19270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c46a6ace434ca3949b816130850824" candidate_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "64118e159acd4c9b89b5267d5d34ee79" is_pre_election: true
I20240503 14:06:27.139307 19150 leader_election.cc:302] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters:
I20240503 14:06:27.139668 19270 raft_consensus.cc:2454] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a in term 1.
I20240503 14:06:27.140197 19313 raft_consensus.cc:2781] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:06:27.140506 19313 raft_consensus.cc:482] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:27.140756 19313 raft_consensus.cc:3037] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:27.144485 19317 raft_consensus.cc:482] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:27.144927 19317 raft_consensus.cc:504] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:27.145879 19313 raft_consensus.cc:504] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.146690 19317 leader_election.cc:288] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481), 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721)
I20240503 14:06:27.147567 19270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64118e159acd4c9b89b5267d5d34ee79" is_pre_election: true
I20240503 14:06:27.147770 19313 leader_election.cc:288] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [CANDIDATE]: Term 2 election: Requested vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
I20240503 14:06:27.148509 19270 raft_consensus.cc:2396] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 14:06:27.147826 19109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd" is_pre_election: true
I20240503 14:06:27.148690 19108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c46a6ace434ca3949b816130850824" candidate_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd"
I20240503 14:06:27.148898 19269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c46a6ace434ca3949b816130850824" candidate_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "64118e159acd4c9b89b5267d5d34ee79"
I20240503 14:06:27.149629 19109 raft_consensus.cc:2454] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a in term 1.
I20240503 14:06:27.149688 19108 raft_consensus.cc:3037] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:27.150188 19269 raft_consensus.cc:3037] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:27.150876 19150 leader_election.cc:302] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters: 64118e159acd4c9b89b5267d5d34ee79
I20240503 14:06:27.151666 19313 raft_consensus.cc:2781] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:06:27.152034 19313 raft_consensus.cc:482] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:27.152308 19313 raft_consensus.cc:3037] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:27.157115 19313 raft_consensus.cc:504] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:27.157541 19269 raft_consensus.cc:2454] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a in term 2.
I20240503 14:06:27.158782 19152 leader_election.cc:302] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64118e159acd4c9b89b5267d5d34ee79, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters:
I20240503 14:06:27.158903 19313 leader_election.cc:288] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [CANDIDATE]: Term 2 election: Requested vote from peers 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481), 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721)
I20240503 14:06:27.158984 19108 raft_consensus.cc:2454] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a in term 2.
I20240503 14:06:27.159672 19313 raft_consensus.cc:2781] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:06:27.160161 19313 raft_consensus.cc:686] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 LEADER]: Becoming Leader. State: Replica: 92dddf617ffe4b4cb9dfb1f5e8a8e29a, State: Running, Role: LEADER
I20240503 14:06:27.160117 19109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd"
I20240503 14:06:27.160935 19109 raft_consensus.cc:3037] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:27.160880 19269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64118e159acd4c9b89b5267d5d34ee79"
I20240503 14:06:27.161739 19269 raft_consensus.cc:3037] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:27.161219 19313 consensus_queue.cc:237] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.167723 19269 raft_consensus.cc:2396] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 14:06:27.168720 19109 raft_consensus.cc:2454] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92dddf617ffe4b4cb9dfb1f5e8a8e29a in term 2.
I20240503 14:06:27.169744 19150 leader_election.cc:302] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters: 64118e159acd4c9b89b5267d5d34ee79
I20240503 14:06:27.169943 18641 catalog_manager.cc:5484] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } health_report { overall_health: UNKNOWN } } }
I20240503 14:06:27.170547 19313 raft_consensus.cc:2781] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:06:27.171185 19313 raft_consensus.cc:686] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 LEADER]: Becoming Leader. State: Replica: 92dddf617ffe4b4cb9dfb1f5e8a8e29a, State: Running, Role: LEADER
I20240503 14:06:27.171835 19313 consensus_queue.cc:237] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:27.182955 18641 catalog_manager.cc:5484] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a reported cstate change: term changed from 1 to 2, leader changed from 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67) to 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66). New cstate: current_term: 2 leader_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } health_report { overall_health: HEALTHY } } }
I20240503 14:06:27.260978 19321 raft_consensus.cc:482] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:27.261865 19321 raft_consensus.cc:504] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.264946 19321 leader_election.cc:288] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:27.280979 19109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577847582a4a19804ec9bab95ad978" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd" is_pre_election: true
I20240503 14:06:27.282066 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577847582a4a19804ec9bab95ad978" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" is_pre_election: true
I20240503 14:06:27.283010 19190 raft_consensus.cc:2454] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 2.
I20240503 14:06:27.284567 19232 leader_election.cc:302] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64118e159acd4c9b89b5267d5d34ee79, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters: 0af02be205214faaaa7f719ee14400bd
I20240503 14:06:27.285365 19321 raft_consensus.cc:2781] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:06:27.285713 19321 raft_consensus.cc:482] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:27.285956 19321 raft_consensus.cc:3037] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:06:27.289876 19304 raft_consensus.cc:482] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:27.290735 19304 raft_consensus.cc:504] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.293776 19304 leader_election.cc:288] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261), 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
I20240503 14:06:27.295184 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2816c401ee24977962c4ebca54ff64f" candidate_uuid: "0af02be205214faaaa7f719ee14400bd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" is_pre_election: true
I20240503 14:06:27.295929 19190 raft_consensus.cc:2454] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0af02be205214faaaa7f719ee14400bd in term 1.
I20240503 14:06:27.295817 19269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2816c401ee24977962c4ebca54ff64f" candidate_uuid: "0af02be205214faaaa7f719ee14400bd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64118e159acd4c9b89b5267d5d34ee79" is_pre_election: true
I20240503 14:06:27.295918 19321 raft_consensus.cc:504] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.297434 19269 raft_consensus.cc:2396] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0af02be205214faaaa7f719ee14400bd for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 14:06:27.297451 19071 leader_election.cc:302] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters:
I20240503 14:06:27.298835 19321 leader_election.cc:288] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 3 election: Requested vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:27.299155 19109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577847582a4a19804ec9bab95ad978" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd"
I20240503 14:06:27.299528 19304 raft_consensus.cc:2781] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:06:27.299918 19304 raft_consensus.cc:482] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:27.300071 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577847582a4a19804ec9bab95ad978" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a"
I20240503 14:06:27.300365 19304 raft_consensus.cc:3037] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:27.300791 19190 raft_consensus.cc:3037] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:06:27.310048 19190 raft_consensus.cc:2454] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 3.
I20240503 14:06:27.310077 19304 raft_consensus.cc:504] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.311594 19232 leader_election.cc:302] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64118e159acd4c9b89b5267d5d34ee79, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters: 0af02be205214faaaa7f719ee14400bd
I20240503 14:06:27.312410 19304 leader_election.cc:288] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [CANDIDATE]: Term 2 election: Requested vote from peers 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261), 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
I20240503 14:06:27.312507 19321 raft_consensus.cc:2781] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:06:27.313647 19269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2816c401ee24977962c4ebca54ff64f" candidate_uuid: "0af02be205214faaaa7f719ee14400bd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64118e159acd4c9b89b5267d5d34ee79"
I20240503 14:06:27.313650 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2816c401ee24977962c4ebca54ff64f" candidate_uuid: "0af02be205214faaaa7f719ee14400bd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a"
I20240503 14:06:27.314298 19321 raft_consensus.cc:686] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 LEADER]: Becoming Leader. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Running, Role: LEADER
I20240503 14:06:27.314567 19269 raft_consensus.cc:3037] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:27.314687 19190 raft_consensus.cc:3037] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:06:27.315958 19321 consensus_queue.cc:237] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.324080 19190 raft_consensus.cc:2454] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0af02be205214faaaa7f719ee14400bd in term 2.
I20240503 14:06:27.325861 19071 leader_election.cc:302] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters:
I20240503 14:06:27.326501 19269 raft_consensus.cc:2396] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0af02be205214faaaa7f719ee14400bd for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 14:06:27.327054 19304 raft_consensus.cc:2781] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:06:27.327672 19304 raft_consensus.cc:686] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 2 LEADER]: Becoming Leader. State: Replica: 0af02be205214faaaa7f719ee14400bd, State: Running, Role: LEADER
I20240503 14:06:27.328756 19304 consensus_queue.cc:237] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.328043 18641 catalog_manager.cc:5484] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 reported cstate change: term changed from 2 to 3, leader changed from 0af02be205214faaaa7f719ee14400bd (127.13.158.65) to 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67). New cstate: current_term: 3 leader_uuid: "64118e159acd4c9b89b5267d5d34ee79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } health_report { overall_health: HEALTHY } } }
I20240503 14:06:27.332895 19321 raft_consensus.cc:482] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:27.333840 19321 raft_consensus.cc:504] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:27.337659 19109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd" is_pre_election: true
I20240503 14:06:27.338544 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" is_pre_election: true
I20240503 14:06:27.338949 19321 leader_election.cc:288] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:27.338636 19109 raft_consensus.cc:2454] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 2.
I20240503 14:06:27.344051 19230 leader_election.cc:302] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 64118e159acd4c9b89b5267d5d34ee79; no voters: 92dddf617ffe4b4cb9dfb1f5e8a8e29a
I20240503 14:06:27.344834 19321 raft_consensus.cc:2781] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:06:27.344123 18643 catalog_manager.cc:5484] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd reported cstate change: term changed from 1 to 2, leader changed from 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67) to 0af02be205214faaaa7f719ee14400bd (127.13.158.65). New cstate: current_term: 2 leader_uuid: "0af02be205214faaaa7f719ee14400bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } health_report { overall_health: UNKNOWN } } }
I20240503 14:06:27.345202 19321 raft_consensus.cc:482] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:27.345645 19321 raft_consensus.cc:3037] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:06:27.352001 19321 raft_consensus.cc:504] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:27.353885 19321 leader_election.cc:288] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 3 election: Requested vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:27.354857 19109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd"
I20240503 14:06:27.355032 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d36677e8ca4144b57786d9bd693fe0" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a"
I20240503 14:06:27.355500 19109 raft_consensus.cc:3037] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:06:27.361641 19109 raft_consensus.cc:2454] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 3.
I20240503 14:06:27.362792 19230 leader_election.cc:302] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0af02be205214faaaa7f719ee14400bd, 64118e159acd4c9b89b5267d5d34ee79; no voters: 92dddf617ffe4b4cb9dfb1f5e8a8e29a
I20240503 14:06:27.363713 19321 raft_consensus.cc:2781] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:06:27.364166 19321 raft_consensus.cc:686] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 LEADER]: Becoming Leader. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Running, Role: LEADER
I20240503 14:06:27.365036 19321 consensus_queue.cc:237] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:27.372443 18643 catalog_manager.cc:5484] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 reported cstate change: term changed from 2 to 3, leader changed from 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66) to 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67). New cstate: current_term: 3 leader_uuid: "64118e159acd4c9b89b5267d5d34ee79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } health_report { overall_health: UNKNOWN } } }
I20240503 14:06:27.429090 19269 raft_consensus.cc:1226] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 LEADER]: Rejecting Update request from peer 0af02be205214faaaa7f719ee14400bd for earlier term 2. Current term is 3. Ops: []
I20240503 14:06:27.430550 19304 consensus_queue.cc:1046] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:06:27.431329 19304 consensus_queue.cc:1225] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [LEADER]: Peer 64118e159acd4c9b89b5267d5d34ee79 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 14:06:27.431919 19305 raft_consensus.cc:3032] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:06:27.432217 19305 raft_consensus.cc:729] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0af02be205214faaaa7f719ee14400bd, State: Running, Role: LEADER
I20240503 14:06:27.433218 19305 consensus_queue.cc:260] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.434410 19305 raft_consensus.cc:3037] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:06:27.552877 19269 raft_consensus.cc:1226] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 LEADER]: Rejecting Update request from peer 92dddf617ffe4b4cb9dfb1f5e8a8e29a for earlier term 2. Current term is 3. Ops: []
I20240503 14:06:27.554370 19313 consensus_queue.cc:1046] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:06:27.554615 19321 raft_consensus.cc:482] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:06:27.555186 19313 consensus_queue.cc:1225] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [LEADER]: Peer 64118e159acd4c9b89b5267d5d34ee79 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 14:06:27.555168 19321 raft_consensus.cc:504] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.555888 19317 raft_consensus.cc:3032] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:06:27.556181 19317 raft_consensus.cc:729] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 92dddf617ffe4b4cb9dfb1f5e8a8e29a, State: Running, Role: LEADER
I20240503 14:06:27.557868 19321 leader_election.cc:288] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:27.557293 19317 consensus_queue.cc:260] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } }
I20240503 14:06:27.558782 19317 raft_consensus.cc:3037] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:06:27.558663 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2816c401ee24977962c4ebca54ff64f" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" is_pre_election: true
I20240503 14:06:27.559446 19109 raft_consensus.cc:1226] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Rejecting Update request from peer 92dddf617ffe4b4cb9dfb1f5e8a8e29a for earlier term 2. Current term is 3. Ops: []
I20240503 14:06:27.559463 19190 raft_consensus.cc:2454] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 2.
I20240503 14:06:27.559738 19108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2816c401ee24977962c4ebca54ff64f" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd" is_pre_election: true
I20240503 14:06:27.561081 19232 leader_election.cc:302] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64118e159acd4c9b89b5267d5d34ee79, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters:
I20240503 14:06:27.562223 19321 raft_consensus.cc:2781] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:06:27.562561 19321 raft_consensus.cc:482] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:06:27.562842 19321 raft_consensus.cc:3037] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:06:27.568179 19321 raft_consensus.cc:504] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.569947 19321 leader_election.cc:288] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 3 election: Requested vote from peers 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721), 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:27.570837 19109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2816c401ee24977962c4ebca54ff64f" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0af02be205214faaaa7f719ee14400bd"
I20240503 14:06:27.570998 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2816c401ee24977962c4ebca54ff64f" candidate_uuid: "64118e159acd4c9b89b5267d5d34ee79" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a"
I20240503 14:06:27.571554 19190 raft_consensus.cc:3037] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:06:27.576710 19190 raft_consensus.cc:2454] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64118e159acd4c9b89b5267d5d34ee79 in term 3.
I20240503 14:06:27.577701 19232 leader_election.cc:302] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64118e159acd4c9b89b5267d5d34ee79, 92dddf617ffe4b4cb9dfb1f5e8a8e29a; no voters: 0af02be205214faaaa7f719ee14400bd
I20240503 14:06:27.578588 19321 raft_consensus.cc:2781] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:06:27.579026 19321 raft_consensus.cc:686] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 3 LEADER]: Becoming Leader. State: Replica: 64118e159acd4c9b89b5267d5d34ee79, State: Running, Role: LEADER
I20240503 14:06:27.579861 19321 consensus_queue.cc:237] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.586694 18643 catalog_manager.cc:5484] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 reported cstate change: term changed from 2 to 3, leader changed from 0af02be205214faaaa7f719ee14400bd (127.13.158.65) to 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67). New cstate: current_term: 3 leader_uuid: "64118e159acd4c9b89b5267d5d34ee79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } health_report { overall_health: HEALTHY } } }
I20240503 14:06:27.740022 19269 raft_consensus.cc:1226] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 3 LEADER]: Rejecting Update request from peer 0af02be205214faaaa7f719ee14400bd for earlier term 2. Current term is 3. Ops: []
I20240503 14:06:27.741284 19305 consensus_queue.cc:1046] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:06:27.741922 19305 consensus_queue.cc:1225] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [LEADER]: Peer 64118e159acd4c9b89b5267d5d34ee79 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 14:06:27.742352 19304 raft_consensus.cc:3032] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:06:27.742666 19304 raft_consensus.cc:729] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0af02be205214faaaa7f719ee14400bd, State: Running, Role: LEADER
I20240503 14:06:27.743342 19304 consensus_queue.cc:260] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } }
I20240503 14:06:27.744220 19304 raft_consensus.cc:3037] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:06:27.755724 19109 raft_consensus.cc:1261] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Refusing update from remote peer 64118e159acd4c9b89b5267d5d34ee79: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 14:06:27.757125 19321 consensus_queue.cc:1035] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:06:27.758080 19321 consensus_queue.cc:1225] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Peer 0af02be205214faaaa7f719ee14400bd log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 14:06:27.758836 19269 raft_consensus.cc:1261] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Refusing update from remote peer 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:06:27.760087 19317 consensus_queue.cc:1035] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:06:27.761061 19109 pending_rounds.cc:76] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: Aborting all ops after (but not including) 0
I20240503 14:06:27.761611 19109 pending_rounds.cc:97] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240503 14:06:27.762009 19109 raft_consensus.cc:2866] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:06:27.790699 19190 raft_consensus.cc:1261] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Refusing update from remote peer 64118e159acd4c9b89b5267d5d34ee79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 14:06:27.792061 19109 raft_consensus.cc:1261] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Refusing update from remote peer 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:06:27.794330 19321 consensus_queue.cc:1035] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:06:27.803452 19335 consensus_queue.cc:1035] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:06:27.835114 19109 raft_consensus.cc:1261] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Refusing update from remote peer 64118e159acd4c9b89b5267d5d34ee79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 14:06:27.836524 19325 consensus_queue.cc:1035] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:06:27.847776 19190 raft_consensus.cc:1261] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Refusing update from remote peer 64118e159acd4c9b89b5267d5d34ee79: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 14:06:27.849678 19325 consensus_queue.cc:1035] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:06:27.850559 19325 consensus_queue.cc:1225] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Peer 92dddf617ffe4b4cb9dfb1f5e8a8e29a log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 14:06:27.853298 19190 pending_rounds.cc:76] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Aborting all ops after (but not including) 0
I20240503 14:06:27.853634 19190 pending_rounds.cc:97] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240503 14:06:27.853914 19190 raft_consensus.cc:2866] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:06:28.036561 19190 raft_consensus.cc:1261] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Refusing update from remote peer 64118e159acd4c9b89b5267d5d34ee79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 14:06:28.037719 19325 consensus_queue.cc:1035] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:06:28.052250 19109 raft_consensus.cc:1261] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Refusing update from remote peer 64118e159acd4c9b89b5267d5d34ee79: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 14:06:28.053800 19338 consensus_queue.cc:1035] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:06:28.054627 19338 consensus_queue.cc:1225] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Peer 0af02be205214faaaa7f719ee14400bd log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 14:06:28.057485 19109 pending_rounds.cc:76] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: Aborting all ops after (but not including) 0
I20240503 14:06:28.057752 19109 pending_rounds.cc:97] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240503 14:06:28.057963 19109 raft_consensus.cc:2866] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:06:28.141626 19190 consensus_queue.cc:237] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } attrs { replace: true } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } }
I20240503 14:06:28.147781 19269 raft_consensus.cc:1261] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Refusing update from remote peer 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:06:28.149705 19333 consensus_queue.cc:1035] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:06:28.153718 19109 raft_consensus.cc:1261] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Refusing update from remote peer 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:06:28.155659 19333 consensus_queue.cc:1035] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:06:28.157714 19333 raft_consensus.cc:2932] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } attrs { replace: true } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } } }
I20240503 14:06:28.159366 19269 raft_consensus.cc:2932] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } attrs { replace: true } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } } }
I20240503 14:06:28.171247 19109 raft_consensus.cc:2932] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } attrs { replace: true } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } } }
I20240503 14:06:28.175124 18642 catalog_manager.cc:5484] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 reported cstate change: config changed from index -1 to 3, NON_VOTER 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68) added. New cstate: current_term: 2 leader_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } attrs { replace: true } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } } }
W20240503 14:06:28.180521 19150 consensus_peers.cc:485] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a -> Peer 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68:41469): Couldn't send request to peer 5737a8388fc14c58a4fb5a3d205d083e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45c46a6ace434ca3949b816130850824. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:28.184093 19270 consensus_queue.cc:237] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } attrs { replace: true } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } }
I20240503 14:06:28.193454 19190 raft_consensus.cc:1261] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Refusing update from remote peer 64118e159acd4c9b89b5267d5d34ee79: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:06:28.195783 19337 consensus_queue.cc:1035] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:06:28.196475 19109 raft_consensus.cc:1261] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Refusing update from remote peer 64118e159acd4c9b89b5267d5d34ee79: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:06:28.198554 19337 consensus_queue.cc:1035] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:06:28.203565 19338 raft_consensus.cc:2932] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } attrs { replace: true } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } } }
W20240503 14:06:28.209848 19230 consensus_peers.cc:485] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 -> Peer 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68:41469): Couldn't send request to peer 5737a8388fc14c58a4fb5a3d205d083e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99d36677e8ca4144b57786d9bd693fe0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:28.209383 19109 raft_consensus.cc:2932] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } attrs { replace: true } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } } }
I20240503 14:06:28.210994 19189 raft_consensus.cc:2932] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } attrs { replace: true } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } } }
I20240503 14:06:28.220355 18641 catalog_manager.cc:5484] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 reported cstate change: config changed from index -1 to 3, NON_VOTER 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68) added. New cstate: current_term: 3 leader_uuid: "64118e159acd4c9b89b5267d5d34ee79" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:06:28.221501 19270 consensus_queue.cc:237] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } attrs { replace: true } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } }
I20240503 14:06:28.230966 19189 raft_consensus.cc:1261] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Refusing update from remote peer 64118e159acd4c9b89b5267d5d34ee79: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240503 14:06:28.232084 19230 consensus_peers.cc:485] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 -> Peer 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68:41469): Couldn't send request to peer 5737a8388fc14c58a4fb5a3d205d083e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26577847582a4a19804ec9bab95ad978. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:28.232379 19109 raft_consensus.cc:1261] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Refusing update from remote peer 64118e159acd4c9b89b5267d5d34ee79: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:06:28.232539 19338 consensus_queue.cc:1035] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:06:28.234092 19337 consensus_queue.cc:1035] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:06:28.241387 19337 raft_consensus.cc:2932] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } attrs { replace: true } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } } }
I20240503 14:06:28.244309 19109 raft_consensus.cc:2932] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } attrs { replace: true } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } } }
I20240503 14:06:28.244616 19189 raft_consensus.cc:2932] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } attrs { replace: true } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } } }
I20240503 14:06:28.258749 18642 catalog_manager.cc:5484] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd reported cstate change: config changed from index -1 to 3, NON_VOTER 5737a8388fc14c58a4fb5a3d205d083e (127.13.158.68) added. New cstate: current_term: 3 leader_uuid: "64118e159acd4c9b89b5267d5d34ee79" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } attrs { replace: true } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } } }
I20240503 14:06:28.654032 19349 ts_tablet_manager.cc:902] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e: Initiating tablet copy from peer 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261)
I20240503 14:06:28.657943 19349 tablet_copy_client.cc:308] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e: tablet copy: Beginning tablet copy session from remote peer at address 127.13.158.66:38261
I20240503 14:06:28.670822 19201 tablet_copy_service.cc:140] P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Received BeginTabletCopySession request for tablet 45c46a6ace434ca3949b816130850824 from peer 5737a8388fc14c58a4fb5a3d205d083e ({username='slave'} at 127.0.0.1:51692)
I20240503 14:06:28.671700 19201 tablet_copy_service.cc:161] P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Beginning new tablet copy session on tablet 45c46a6ace434ca3949b816130850824 from peer 5737a8388fc14c58a4fb5a3d205d083e at {username='slave'} at 127.0.0.1:51692: session id = 5737a8388fc14c58a4fb5a3d205d083e-45c46a6ace434ca3949b816130850824
I20240503 14:06:28.679033 19201 tablet_copy_source_session.cc:205] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:06:28.682729 19349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c46a6ace434ca3949b816130850824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:06:28.697666 19349 tablet_copy_client.cc:783] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e: tablet copy: Starting download of 0 data blocks...
I20240503 14:06:28.698323 19349 tablet_copy_client.cc:647] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:06:28.701797 19349 tablet_copy_client.cc:523] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:06:28.710026 19349 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e: Bootstrap starting.
I20240503 14:06:28.727419 19349 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:06:28.728343 19349 tablet_bootstrap.cc:492] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e: Bootstrap complete.
I20240503 14:06:28.729087 19349 ts_tablet_manager.cc:1367] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.007s
I20240503 14:06:28.731679 19349 raft_consensus.cc:348] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } attrs { replace: true } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } }
I20240503 14:06:28.732550 19349 raft_consensus.cc:729] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5737a8388fc14c58a4fb5a3d205d083e, State: Initialized, Role: LEARNER
I20240503 14:06:28.733139 19349 consensus_queue.cc:260] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0af02be205214faaaa7f719ee14400bd" member_type: VOTER last_known_addr { host: "127.13.158.65" port: 40721 } } peers { permanent_uuid: "92dddf617ffe4b4cb9dfb1f5e8a8e29a" member_type: VOTER last_known_addr { host: "127.13.158.66" port: 38261 } } peers { permanent_uuid: "64118e159acd4c9b89b5267d5d34ee79" member_type: VOTER last_known_addr { host: "127.13.158.67" port: 37481 } attrs { replace: true } } peers { permanent_uuid: "5737a8388fc14c58a4fb5a3d205d083e" member_type: NON_VOTER last_known_addr { host: "127.13.158.68" port: 41469 } attrs { promote: true } }
I20240503 14:06:28.735839 19349 ts_tablet_manager.cc:1398] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20240503 14:06:28.737421 19201 tablet_copy_service.cc:342] P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: Request end of tablet copy session 5737a8388fc14c58a4fb5a3d205d083e-45c46a6ace434ca3949b816130850824 received from {username='slave'} at 127.0.0.1:51692
I20240503 14:06:28.737835 19201 tablet_copy_service.cc:433] P 92dddf617ffe4b4cb9dfb1f5e8a8e29a: ending tablet copy session 5737a8388fc14c58a4fb5a3d205d083e-45c46a6ace434ca3949b816130850824 on tablet 45c46a6ace434ca3949b816130850824 with peer 5737a8388fc14c58a4fb5a3d205d083e
I20240503 14:06:28.778034 19349 ts_tablet_manager.cc:902] T 99d36677e8ca4144b57786d9bd693fe0 P 5737a8388fc14c58a4fb5a3d205d083e: Initiating tablet copy from peer 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
F20240503 14:06:28.778944 19349 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1b3fa6230957409db8eff9dccbf62b32.queue_time_us, tablet-download-pool-1b3fa6230957409db8eff9dccbf62b32.queue_time_us) (tablet-download-pool-1b3fa6230957409db8eff9dccbf62b32.run_cpu_time_us, tablet-download-pool-1b3fa6230957409db8eff9dccbf62b32.run_cpu_time_us) (tablet-download-pool-1b3fa6230957409db8eff9dccbf62b32.run_wall_time_us, tablet-download-pool-1b3fa6230957409db8eff9dccbf62b32.run_wall_time_us) (tablet-download-pool-2032d05f926d4989ba2d00c1ea704f8a.queue_time_us, tablet-download-pool-2032d05f926d4989ba2d00c1ea704f8a.queue_time_us) (tablet-download-pool-2032d05f926d4989ba2d00c1ea704f8a.run_cpu_time_us, tablet-download-pool-2032d05f926d4989ba2d00c1ea704f8a.run_cpu_time_us) (tablet-download-pool-2032d05f926d4989ba2d00c1ea704f8a.run_wall_time_us, tablet-download-pool-2032d05f926d4989ba2d00c1ea704f8a.run_wall_time_us) (tablet-download-pool-2540f6226da9468195f0ded09c888a6b.queue_time_us, tablet-download-pool-2540f6226da9468195f0ded09c888a6b.queue_time_us) (tablet-download-pool-2540f6226da9468195f0ded09c888a6b.run_cpu_time_us, tablet-download-pool-2540f6226da9468195f0ded09c888a6b.run_cpu_time_us) (tablet-download-pool-2540f6226da9468195f0ded09c888a6b.run_wall_time_us, tablet-download-pool-2540f6226da9468195f0ded09c888a6b.run_wall_time_us) (tablet-download-pool-2f053f9a1b9e41f68fc778572aaf64cf.queue_time_us, tablet-download-pool-2f053f9a1b9e41f68fc778572aaf64cf.queue_time_us) (tablet-download-pool-2f053f9a1b9e41f68fc778572aaf64cf.run_cpu_time_us, tablet-download-pool-2f053f9a1b9e41f68fc778572aaf64cf.run_cpu_time_us) (tablet-download-pool-2f053f9a1b9e41f68fc778572aaf64cf.run_wall_time_us, tablet-download-pool-2f053f9a1b9e41f68fc778572aaf64cf.run_wall_time_us) (tablet-download-pool-45c46a6ace434ca3949b816130850824.queue_time_us, tablet-download-pool-45c46a6ace434ca3949b816130850824.queue_time_us) (tablet-download-pool-45c46a6ace434ca3949b816130850824.run_cpu_time_us, tablet-download-pool-45c46a6ace434ca3949b816130850824.run_cpu_time_us) (tablet-download-pool-45c46a6ace434ca3949b816130850824.run_wall_time_us, tablet-download-pool-45c46a6ace434ca3949b816130850824.run_wall_time_us) (tablet-download-pool-4e478e48cb48476e9cde5fd215be0e02.queue_time_us, tablet-download-pool-4e478e48cb48476e9cde5fd215be0e02.queue_time_us) (tablet-download-pool-4e478e48cb48476e9cde5fd215be0e02.run_cpu_time_us, tablet-download-pool-4e478e48cb48476e9cde5fd215be0e02.run_cpu_time_us) (tablet-download-pool-4e478e48cb48476e9cde5fd215be0e02.run_wall_time_us, tablet-download-pool-4e478e48cb48476e9cde5fd215be0e02.run_wall_time_us) (tablet-download-pool-72ea0d3526a2405c912b70848da3dbe7.queue_time_us, tablet-download-pool-72ea0d3526a2405c912b70848da3dbe7.queue_time_us) (tablet-download-pool-72ea0d3526a2405c912b70848da3dbe7.run_cpu_time_us, tablet-download-pool-72ea0d3526a2405c912b70848da3dbe7.run_cpu_time_us) (tablet-download-pool-72ea0d3526a2405c912b70848da3dbe7.run_wall_time_us, tablet-download-pool-72ea0d3526a2405c912b70848da3dbe7.run_wall_time_us) (tablet-download-pool-99d36677e8ca4144b57786d9bd693fe0.queue_time_us, tablet-download-pool-99d36677e8ca4144b57786d9bd693fe0.queue_time_us) (tablet-download-pool-99f1c0f5f2354dacbd45ac12c93e70bc.queue_time_us, tablet-download-pool-99f1c0f5f2354dacbd45ac12c93e70bc.queue_time_us) (tablet-download-pool-99f1c0f5f2354dacbd45ac12c93e70bc.run_cpu_time_us, tablet-download-pool-99f1c0f5f2354dacbd45ac12c93e70bc.run_cpu_time_us) (tablet-download-pool-99f1c0f5f2354dacbd45ac12c93e70bc.run_wall_time_us, tablet-download-pool-99f1c0f5f2354dacbd45ac12c93e70bc.run_wall_time_us) (tablet-download-pool-c372eadbe3de4dda8c100977d1f017a7.queue_time_us, tablet-download-pool-c372eadbe3de4dda8c100977d1f017a7.queue_time_us) (tablet-download-pool-c372eadbe3de4dda8c100977d1f017a7.run_cpu_time_us, tablet-download-pool-c372eadbe3de4dda8c100977d1f017a7.run_cpu_time_us) (tablet-download-pool-c372eadbe3de4dda8c100977d1f017a7.run_wall_time_us, tablet-download-pool-c372eadbe3de4dda8c100977d1f017a7.run_wall_time_us) (tablet-download-pool-d2302ac8c2e544c68cf11427f3bf9143.queue_time_us, tablet-download-pool-d2302ac8c2e544c68cf11427f3bf9143.queue_time_us) (tablet-download-pool-d2302ac8c2e544c68cf11427f3bf9143.run_cpu_time_us, tablet-download-pool-d2302ac8c2e544c68cf11427f3bf9143.run_cpu_time_us) (tablet-download-pool-d2302ac8c2e544c68cf11427f3bf9143.run_wall_time_us, tablet-download-pool-d2302ac8c2e544c68cf11427f3bf9143.run_wall_time_us) (tablet-download-pool-da79a01e920f448384284e3d77da4293.queue_time_us, tablet-download-pool-da79a01e920f448384284e3d77da4293.queue_time_us) (tablet-download-pool-da79a01e920f448384284e3d77da4293.run_cpu_time_us, tablet-download-pool-da79a01e920f448384284e3d77da4293.run_cpu_time_us) (tablet-download-pool-da79a01e920f448384284e3d77da4293.run_wall_time_us, tablet-download-pool-da79a01e920f448384284e3d77da4293.run_wall_time_us) (tablet-download-pool-f023e4f1e70c47caad92f740e24b9acc.queue_time_us, tablet-download-pool-f023e4f1e70c47caad92f740e24b9acc.queue_time_us) (tablet-download-pool-f023e4f1e70c47caad92f740e24b9acc.run_cpu_time_us, tablet-download-pool-f023e4f1e70c47caad92f740e24b9acc.run_cpu_time_us) (tablet-download-pool-f023e4f1e70c47caad92f740e24b9acc.run_wall_time_us, tablet-download-pool-f023e4f1e70c47caad92f740e24b9acc.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714745188 (unix time) try "date -d @1714745188" if you are using GNU date ***
I20240503 14:06:28.794488 19358 ts_tablet_manager.cc:902] T 26577847582a4a19804ec9bab95ad978 P 5737a8388fc14c58a4fb5a3d205d083e: Initiating tablet copy from peer 64118e159acd4c9b89b5267d5d34ee79 (127.13.158.67:37481)
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800003679) received by PID 13945 (TID 0x7f2596cfb700) from PID 13945; stack trace: ***
@ 0x2e41a0 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e6683 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f2609615980 (unknown) at ??:0
@ 0x7f2602b89fb7 gsignal at ??:0
@ 0x7f2602b8b921 abort at ??:0
@ 0x2ea196 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f260597fda4 google::LogMessage::SendToLog() at ??:0
@ 0x7f2605980950 google::LogMessage::Flush() at ??:0
@ 0x7f2605985a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f260630d3c7 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f26062e394f kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f26062e10a5 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f260e101422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 14:06:29.080582 19018 raft_consensus.cc:1203] T 45c46a6ace434ca3949b816130850824 P 5737a8388fc14c58a4fb5a3d205d083e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
@ 0x7f260e107c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f260e19d4ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f260e1ce3cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f260e1b41ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240503 14:06:29.157634 13945 tablet_server.cc:178] TabletServer@127.13.158.65:40721 shutting down...
W20240503 14:06:29.181329 19150 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 8 similar messages]
@ 0x7f260e1b417a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
W20240503 14:06:29.213209 19150 consensus_peers.cc:485] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a -> Peer 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721): Couldn't send request to peer 0af02be205214faaaa7f719ee14400bd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:29.216557 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:06:29.217875 13945 tablet_replica.cc:330] stopping tablet replica
I20240503 14:06:29.219019 13945 raft_consensus.cc:2229] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:29.219800 13945 raft_consensus.cc:2258] T 45c46a6ace434ca3949b816130850824 P 0af02be205214faaaa7f719ee14400bd [term 2 FOLLOWER]: Raft consensus is shut down!
@ 0x7f260e1b410a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20240503 14:06:29.223671 13945 tablet_replica.cc:330] stopping tablet replica
I20240503 14:06:29.224522 13945 raft_consensus.cc:2229] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:29.225232 13945 raft_consensus.cc:2258] T 99d36677e8ca4144b57786d9bd693fe0 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:29.228849 13945 tablet_replica.cc:330] stopping tablet replica
I20240503 14:06:29.230083 13945 raft_consensus.cc:2229] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:29.230789 13945 raft_consensus.cc:2258] T 26577847582a4a19804ec9bab95ad978 P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:29.234278 13945 tablet_replica.cc:330] stopping tablet replica
I20240503 14:06:29.234898 13945 raft_consensus.cc:2229] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:29.235392 13945 raft_consensus.cc:2258] T e2816c401ee24977962c4ebca54ff64f P 0af02be205214faaaa7f719ee14400bd [term 3 FOLLOWER]: Raft consensus is shut down!
@ 0x7f260e1b40d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20240503 14:06:29.264647 13945 tablet_server.cc:194] TabletServer@127.13.158.65:40721 shutdown complete.
@ 0x7f260e1b32fe std::__1::__function::__func<>::operator()() at ??:0
I20240503 14:06:29.285538 13945 tablet_server.cc:178] TabletServer@127.13.158.66:38261 shutting down...
W20240503 14:06:29.301784 18691 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 14:06:29.302193 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
@ 0x7f260f559da5 std::__1::__function::__value_func<>::operator()() at ??:0
I20240503 14:06:29.341064 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240503 14:06:29.341398 19232 consensus_peers.cc:485] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 -> Peer 92dddf617ffe4b4cb9dfb1f5e8a8e29a (127.13.158.66:38261): Couldn't send request to peer 92dddf617ffe4b4cb9dfb1f5e8a8e29a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:29.342254 13945 tablet_replica.cc:330] stopping tablet replica
I20240503 14:06:29.343394 13945 raft_consensus.cc:2229] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 LEADER]: Raft consensus shutting down.
I20240503 14:06:29.344815 13945 raft_consensus.cc:2258] T 45c46a6ace434ca3949b816130850824 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:29.347882 13945 tablet_replica.cc:330] stopping tablet replica
I20240503 14:06:29.348678 13945 raft_consensus.cc:2229] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:29.349637 13945 raft_consensus.cc:2258] T 99d36677e8ca4144b57786d9bd693fe0 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:29.352481 13945 tablet_replica.cc:330] stopping tablet replica
I20240503 14:06:29.353207 13945 raft_consensus.cc:2229] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:29.353811 13945 raft_consensus.cc:2258] T 26577847582a4a19804ec9bab95ad978 P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:29.356464 13945 tablet_replica.cc:330] stopping tablet replica
I20240503 14:06:29.357126 13945 raft_consensus.cc:2229] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:29.357651 13945 raft_consensus.cc:2258] T e2816c401ee24977962c4ebca54ff64f P 92dddf617ffe4b4cb9dfb1f5e8a8e29a [term 3 FOLLOWER]: Raft consensus is shut down!
@ 0x7f260f559bda std::__1::function<>::operator()() at ??:0
W20240503 14:06:29.379951 19230 consensus_peers.cc:485] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 -> Peer 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721): Couldn't send request to peer 0af02be205214faaaa7f719ee14400bd. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:40721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f26062e5237 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 14:06:29.388581 13945 tablet_server.cc:194] TabletServer@127.13.158.66:38261 shutdown complete.
W20240503 14:06:29.398133 19230 consensus_peers.cc:485] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 -> Peer 0af02be205214faaaa7f719ee14400bd (127.13.158.65:40721): Couldn't send request to peer 0af02be205214faaaa7f719ee14400bd. Status: Network error: Client connection negotiation failed: client connection to 127.13.158.65:40721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:06:29.412010 13945 tablet_server.cc:178] TabletServer@127.13.158.67:37481 shutting down...
@ 0x7f26062e8692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
W20240503 14:06:29.420225 18691 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 14:06:29.420532 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 14:06:29.427392 18691 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 14:06:29.427779 18691 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
@ 0x7f26062e864a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7f26062e85da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20240503 14:06:29.473732 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:06:29.475008 13945 tablet_replica.cc:330] stopping tablet replica
I20240503 14:06:29.475780 13945 raft_consensus.cc:2229] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:06:29.476719 13945 raft_consensus.cc:2258] T 45c46a6ace434ca3949b816130850824 P 64118e159acd4c9b89b5267d5d34ee79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:29.479593 13945 tablet_replica.cc:330] stopping tablet replica
I20240503 14:06:29.480274 13945 raft_consensus.cc:2229] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 LEADER]: Raft consensus shutting down.
I20240503 14:06:29.481429 13945 raft_consensus.cc:2258] T 99d36677e8ca4144b57786d9bd693fe0 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:29.484592 13945 tablet_replica.cc:330] stopping tablet replica
I20240503 14:06:29.485301 13945 raft_consensus.cc:2229] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 LEADER]: Raft consensus shutting down.
I20240503 14:06:29.486510 13945 raft_consensus.cc:2258] T 26577847582a4a19804ec9bab95ad978 P 64118e159acd4c9b89b5267d5d34ee79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:06:29.489114 13945 tablet_replica.cc:330] stopping tablet replica
I20240503 14:06:29.489784 13945 raft_consensus.cc:2229] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 3 LEADER]: Raft consensus shutting down.
I20240503 14:06:29.490788 13945 raft_consensus.cc:2258] T e2816c401ee24977962c4ebca54ff64f P 64118e159acd4c9b89b5267d5d34ee79 [term 3 FOLLOWER]: Raft consensus is shut down!
@ 0x7f26062e85a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f26062e789e std::__1::__function::__func<>::operator()() at ??:0
I20240503 14:06:29.520258 13945 tablet_server.cc:194] TabletServer@127.13.158.67:37481 shutdown complete.
@ 0x7f260f559da5 std::__1::__function::__value_func<>::operator()() at ??:0
I20240503 14:06:29.539351 13945 tablet_server.cc:178] TabletServer@127.13.158.68:0 shutting down...
@ 0x7f260f559bda std::__1::function<>::operator()() at ??:0
I20240503 14:06:29.578573 13945 ts_tablet_manager.cc:1470] Shutting down tablet manager...
@ 0x7f26062c7937 kudu::Thread::SuperviseThread() at ??:0