[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 21:25:26.111311 25665 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.16.126:33669,127.25.16.125:45259,127.25.16.124:34957
I20240502 21:25:26.113075 25665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:26.114235 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:26.125100 25673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:26.125056 25671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:26.125069 25670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.125717 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:26.126526 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:26.126698 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:26.126770 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685126126749 us; error 0 us; skew 500 ppm
I20240502 21:25:26.127049 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:26.132493 25665 webserver.cc:460] Webserver started at http://127.25.16.126:38723/ using document root <none> and password file <none>
I20240502 21:25:26.133172 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:26.133261 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:26.133556 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:26.136826 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-0-root/instance:
uuid: "8984b42adf0a49e8ae78ec1755f26349"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-b524"
I20240502 21:25:26.142374 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240502 21:25:26.145709 25678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.147249 25665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:25:26.147441 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-0-root
uuid: "8984b42adf0a49e8ae78ec1755f26349"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-b524"
I20240502 21:25:26.147627 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:26.174997 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:26.175871 25665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:26.176160 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:26.193838 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.126:33669
I20240502 21:25:26.193876 25729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.126:33669 every 8 connection(s)
I20240502 21:25:26.195986 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:26.198765 25732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:26.198859 25733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.199417 25730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.200542 25665 server_base.cc:1034] running on GCE node
W20240502 21:25:26.200803 25735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.201253 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:26.201344 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:26.201460 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685126201418 us; error 0 us; skew 500 ppm
I20240502 21:25:26.201737 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:26.204751 25665 webserver.cc:460] Webserver started at http://127.25.16.125:41281/ using document root <none> and password file <none>
I20240502 21:25:26.205039 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:26.205144 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:26.205300 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:26.206117 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-1-root/instance:
uuid: "a44dde6cefe642149ef3bb78cbc98d79"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-b524"
I20240502 21:25:26.209458 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:25:26.211577 25740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.211951 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 21:25:26.212064 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-1-root
uuid: "a44dde6cefe642149ef3bb78cbc98d79"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-b524"
I20240502 21:25:26.212189 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:26.213232 25730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } has no permanent_uuid. Determining permanent_uuid...
I20240502 21:25:26.223250 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:26.223867 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:26.232304 25730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } has no permanent_uuid. Determining permanent_uuid...
W20240502 21:25:26.233811 25680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.16.125:45259: connect: Connection refused (error 111)
W20240502 21:25:26.235813 25730 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.25.16.125:45259: Network error: Client connection negotiation failed: client connection to 127.25.16.125:45259: connect: Connection refused (error 111)
I20240502 21:25:26.242376 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.125:45259
I20240502 21:25:26.242413 25794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.125:45259 every 8 connection(s)
I20240502 21:25:26.243808 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 21:25:26.243976 25795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:25:26.246941 25797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.247756 25795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } has no permanent_uuid. Determining permanent_uuid...
W20240502 21:25:26.248499 25798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:26.251477 25800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.252131 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:26.252542 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:26.252683 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:26.252770 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685126252727 us; error 0 us; skew 500 ppm
I20240502 21:25:26.252948 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:26.254542 25665 webserver.cc:460] Webserver started at http://127.25.16.124:40113/ using document root <none> and password file <none>
I20240502 21:25:26.254889 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:26.255100 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:26.255306 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:26.256363 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-2-root/instance:
uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-b524"
I20240502 21:25:26.257648 25795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } has no permanent_uuid. Determining permanent_uuid...
I20240502 21:25:26.259987 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:25:26.262315 25807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.262802 25665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.262943 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-2-root
uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-b524"
I20240502 21:25:26.263185 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:26.265650 25795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } has no permanent_uuid. Determining permanent_uuid...
W20240502 21:25:26.268144 25795 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.25.16.124:34957: Network error: Client connection negotiation failed: client connection to 127.25.16.124:34957: connect: Connection refused (error 111)
I20240502 21:25:26.284864 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:26.285187 25730 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } attempt: 1
I20240502 21:25:26.285799 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:26.292090 25730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } has no permanent_uuid. Determining permanent_uuid...
W20240502 21:25:26.294025 25730 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.25.16.124:34957: Network error: Client connection negotiation failed: client connection to 127.25.16.124:34957: connect: Connection refused (error 111)
I20240502 21:25:26.309650 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.124:34957
I20240502 21:25:26.309716 25859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.124:34957 every 8 connection(s)
I20240502 21:25:26.310703 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:25:26.311139 25860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.314148 25860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } has no permanent_uuid. Determining permanent_uuid...
I20240502 21:25:26.320348 25795 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } attempt: 1
I20240502 21:25:26.321429 25860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } has no permanent_uuid. Determining permanent_uuid...
I20240502 21:25:26.327591 25860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } has no permanent_uuid. Determining permanent_uuid...
I20240502 21:25:26.332316 25795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79: Bootstrap starting.
I20240502 21:25:26.335390 25860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c: Bootstrap starting.
I20240502 21:25:26.336174 25795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.336702 25860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.337240 25730 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } attempt: 1
I20240502 21:25:26.337656 25860 log.cc:824] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:26.340700 25795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79: No bootstrap required, opened a new log
I20240502 21:25:26.340691 25860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c: No bootstrap required, opened a new log
I20240502 21:25:26.346341 25730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349: Bootstrap starting.
I20240502 21:25:26.347869 25730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.349897 25730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349: No bootstrap required, opened a new log
I20240502 21:25:26.350476 25860 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } }
I20240502 21:25:26.350450 25795 raft_consensus.cc:348] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } }
I20240502 21:25:26.350647 25730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } }
I20240502 21:25:26.350788 25860 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.350786 25795 raft_consensus.cc:374] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.350891 25730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.350975 25795 raft_consensus.cc:729] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44dde6cefe642149ef3bb78cbc98d79, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.351001 25730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8984b42adf0a49e8ae78ec1755f26349, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.350993 25860 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d2c3d65b7c4b1e86edce8e6242fb9c, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.351756 25860 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } }
I20240502 21:25:26.351828 25795 consensus_queue.cc:260] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } }
I20240502 21:25:26.351786 25730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } }
I20240502 21:25:26.352842 25868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } } }
I20240502 21:25:26.353245 25868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:26.352869 25867 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } } }
I20240502 21:25:26.353426 25867 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:26.354036 25730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:26.354044 25860 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:26.361080 25795 sys_catalog.cc:564] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:26.360905 25869 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } } }
I20240502 21:25:26.361227 25869 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:26.371685 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240502 21:25:26.380803 25894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 21:25:26.381016 25890 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 21:25:26.381429 25890 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 21:25:26.381310 25894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 21:25:26.385802 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 21:25:26.386726 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:26.387332 25903 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 21:25:26.387440 25903 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 21:25:26.390462 25904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:26.390743 25905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:26.391477 25907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.392385 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:26.392648 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:26.392733 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:26.392793 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685126392773 us; error 0 us; skew 500 ppm
I20240502 21:25:26.392971 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:26.394354 25665 webserver.cc:460] Webserver started at http://127.25.16.65:38387/ using document root <none> and password file <none>
I20240502 21:25:26.394634 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:26.394728 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:26.394899 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:26.395583 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-0-root/instance:
uuid: "d303291ceac849de920a5e02566742c8"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-b524"
I20240502 21:25:26.398751 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:25:26.400703 25912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.401094 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 21:25:26.401206 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-0-root
uuid: "d303291ceac849de920a5e02566742c8"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-b524"
I20240502 21:25:26.401329 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:26.410559 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:26.411166 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:26.411981 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:26.413703 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:26.413829 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.413944 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:26.414021 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20240502 21:25:26.428359 25684 tablet.cc:2369] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:26.435292 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.65:39061
I20240502 21:25:26.435330 25974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.65:39061 every 8 connection(s)
I20240502 21:25:26.448925 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 21:25:26.468452 25977 heartbeater.cc:347] Connected to a master server at 127.25.16.125:45259
I20240502 21:25:26.468767 25977 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:26.470000 25977 heartbeater.cc:511] Master 127.25.16.125:45259 requested a full tablet report, sending...
I20240502 21:25:26.472944 25759 ts_manager.cc:194] Registered new tserver with Master: d303291ceac849de920a5e02566742c8 (127.25.16.65:39061)
I20240502 21:25:26.477201 25976 heartbeater.cc:347] Connected to a master server at 127.25.16.126:33669
I20240502 21:25:26.477346 25976 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:26.478169 25976 heartbeater.cc:511] Master 127.25.16.126:33669 requested a full tablet report, sending...
I20240502 21:25:26.479439 25695 ts_manager.cc:194] Registered new tserver with Master: d303291ceac849de920a5e02566742c8 (127.25.16.65:39061)
W20240502 21:25:26.481992 25982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:26.482455 25983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.482522 25975 heartbeater.cc:347] Connected to a master server at 127.25.16.124:34957
I20240502 21:25:26.482684 25975 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:26.483279 25975 heartbeater.cc:511] Master 127.25.16.124:34957 requested a full tablet report, sending...
W20240502 21:25:26.484450 25985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.484717 25825 ts_manager.cc:194] Registered new tserver with Master: d303291ceac849de920a5e02566742c8 (127.25.16.65:39061)
I20240502 21:25:26.489527 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:26.489820 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:26.489888 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:26.489960 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685126489924 us; error 0 us; skew 500 ppm
I20240502 21:25:26.490128 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:26.491400 25665 webserver.cc:460] Webserver started at http://127.25.16.66:34713/ using document root <none> and password file <none>
I20240502 21:25:26.491627 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:26.491727 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:26.491866 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:26.492537 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-1-root/instance:
uuid: "7c29cdbaf98d424fb6cb46a5609d60ad"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-b524"
I20240502 21:25:26.495121 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.496951 25990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.497309 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 21:25:26.497426 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-1-root
uuid: "7c29cdbaf98d424fb6cb46a5609d60ad"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-b524"
I20240502 21:25:26.497550 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:26.516666 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:26.517282 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:26.517872 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:26.518872 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:26.518947 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.519073 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:26.519129 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.538190 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.66:35489
I20240502 21:25:26.538210 26052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.66:35489 every 8 connection(s)
I20240502 21:25:26.558789 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 21:25:26.559815 26053 heartbeater.cc:347] Connected to a master server at 127.25.16.124:34957
I20240502 21:25:26.559955 26053 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:26.560639 26053 heartbeater.cc:511] Master 127.25.16.124:34957 requested a full tablet report, sending...
I20240502 21:25:26.561975 25825 ts_manager.cc:194] Registered new tserver with Master: 7c29cdbaf98d424fb6cb46a5609d60ad (127.25.16.66:35489)
I20240502 21:25:26.564069 26054 heartbeater.cc:347] Connected to a master server at 127.25.16.126:33669
I20240502 21:25:26.564211 26054 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:26.564908 26054 heartbeater.cc:511] Master 127.25.16.126:33669 requested a full tablet report, sending...
I20240502 21:25:26.566149 25695 ts_manager.cc:194] Registered new tserver with Master: 7c29cdbaf98d424fb6cb46a5609d60ad (127.25.16.66:35489)
I20240502 21:25:26.568199 26055 heartbeater.cc:347] Connected to a master server at 127.25.16.125:45259
I20240502 21:25:26.568439 26055 heartbeater.cc:464] Registering TS with master...
W20240502 21:25:26.568888 26059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.569200 26055 heartbeater.cc:511] Master 127.25.16.125:45259 requested a full tablet report, sending...
W20240502 21:25:26.569428 26060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.570299 25759 ts_manager.cc:194] Registered new tserver with Master: 7c29cdbaf98d424fb6cb46a5609d60ad (127.25.16.66:35489)
I20240502 21:25:26.573983 25665 server_base.cc:1034] running on GCE node
W20240502 21:25:26.576553 26063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:26.576970 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:26.577066 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:26.577126 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685126577103 us; error 0 us; skew 500 ppm
I20240502 21:25:26.577296 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:26.578788 25665 webserver.cc:460] Webserver started at http://127.25.16.67:35569/ using document root <none> and password file <none>
I20240502 21:25:26.579027 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:26.579114 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:26.579258 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:26.579913 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-2-root/instance:
uuid: "5efb33455c734d2599b359f95a194ffb"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-b524"
I20240502 21:25:26.582777 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:26.584753 26067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.585180 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 21:25:26.585289 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-2-root
uuid: "5efb33455c734d2599b359f95a194ffb"
format_stamp: "Formatted at 2024-05-02 21:25:26 on dist-test-slave-b524"
I20240502 21:25:26.585397 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:26.601699 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:26.602380 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:26.603034 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:26.603968 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:26.604069 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.604187 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:26.604293 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.604403 25868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.604699 25868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } }
I20240502 21:25:26.606946 25868 leader_election.cc:288] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44dde6cefe642149ef3bb78cbc98d79 (127.25.16.125:45259), b2d2c3d65b7c4b1e86edce8e6242fb9c (127.25.16.124:34957)
I20240502 21:25:26.607650 25770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8984b42adf0a49e8ae78ec1755f26349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44dde6cefe642149ef3bb78cbc98d79" is_pre_election: true
I20240502 21:25:26.607748 25835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8984b42adf0a49e8ae78ec1755f26349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" is_pre_election: true
I20240502 21:25:26.608150 25770 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8984b42adf0a49e8ae78ec1755f26349 in term 0.
I20240502 21:25:26.608155 25835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8984b42adf0a49e8ae78ec1755f26349 in term 0.
I20240502 21:25:26.608855 25680 leader_election.cc:302] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8984b42adf0a49e8ae78ec1755f26349, a44dde6cefe642149ef3bb78cbc98d79; no voters:
I20240502 21:25:26.609309 25868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.609508 25868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.609617 25868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.613022 25868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } }
I20240502 21:25:26.614598 25770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8984b42adf0a49e8ae78ec1755f26349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44dde6cefe642149ef3bb78cbc98d79"
I20240502 21:25:26.614885 25770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.618331 25770 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8984b42adf0a49e8ae78ec1755f26349 in term 1.
I20240502 21:25:26.618733 25868 leader_election.cc:288] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [CANDIDATE]: Term 1 election: Requested vote from peers a44dde6cefe642149ef3bb78cbc98d79 (127.25.16.125:45259), b2d2c3d65b7c4b1e86edce8e6242fb9c (127.25.16.124:34957)
I20240502 21:25:26.618817 25680 leader_election.cc:302] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8984b42adf0a49e8ae78ec1755f26349, a44dde6cefe642149ef3bb78cbc98d79; no voters:
I20240502 21:25:26.619154 25868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.619832 25835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8984b42adf0a49e8ae78ec1755f26349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c"
I20240502 21:25:26.620035 25835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.620677 25868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 1 LEADER]: Becoming Leader. State: Replica: 8984b42adf0a49e8ae78ec1755f26349, State: Running, Role: LEADER
I20240502 21:25:26.621469 25868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } }
I20240502 21:25:26.623585 25835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8984b42adf0a49e8ae78ec1755f26349 in term 1.
I20240502 21:25:26.626601 26083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8984b42adf0a49e8ae78ec1755f26349. Latest consensus state: current_term: 1 leader_uuid: "8984b42adf0a49e8ae78ec1755f26349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } } }
I20240502 21:25:26.626971 26083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:26.632825 26088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:26.634804 26088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:26.645620 25770 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [term 1 FOLLOWER]: Refusing update from remote peer 8984b42adf0a49e8ae78ec1755f26349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:26.645628 25835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [term 1 FOLLOWER]: Refusing update from remote peer 8984b42adf0a49e8ae78ec1755f26349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:26.646668 26083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:26.646946 25868 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:26.667742 25869 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8984b42adf0a49e8ae78ec1755f26349. Latest consensus state: current_term: 1 leader_uuid: "8984b42adf0a49e8ae78ec1755f26349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } } }
I20240502 21:25:26.668148 25869 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:26.668815 25867 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8984b42adf0a49e8ae78ec1755f26349. Latest consensus state: current_term: 1 leader_uuid: "8984b42adf0a49e8ae78ec1755f26349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } } }
I20240502 21:25:26.669118 25867 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:26.672425 26091 mvcc.cc:205] Tried to move back new op lower bound from 7023350278727626752 to 7023350278647767040. Current Snapshot: MvccSnapshot[applied={T|T < 7023350278727626752}]
I20240502 21:25:26.681847 26083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8984b42adf0a49e8ae78ec1755f26349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } } }
I20240502 21:25:26.682119 26083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:26.684504 26088 catalog_manager.cc:1347] Generated new cluster ID: 5c2233d96d81481ea85062a9bd7d70fb
I20240502 21:25:26.684626 26088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:26.690385 25869 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8984b42adf0a49e8ae78ec1755f26349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } } }
I20240502 21:25:26.690599 25869 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:26.699611 26083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8984b42adf0a49e8ae78ec1755f26349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } } }
I20240502 21:25:26.699908 26083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:26.712548 26088 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:26.714395 26088 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:26.725075 25867 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8984b42adf0a49e8ae78ec1755f26349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8984b42adf0a49e8ae78ec1755f26349" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 33669 } } peers { permanent_uuid: "a44dde6cefe642149ef3bb78cbc98d79" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 45259 } } peers { permanent_uuid: "b2d2c3d65b7c4b1e86edce8e6242fb9c" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 34957 } } }
I20240502 21:25:26.725330 25867 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:26.732123 26088 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349: Generated new TSK 0
I20240502 21:25:26.733287 26088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:26.746474 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.67:37545
I20240502 21:25:26.746526 26141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.67:37545 every 8 connection(s)
I20240502 21:25:26.771695 26142 heartbeater.cc:347] Connected to a master server at 127.25.16.124:34957
I20240502 21:25:26.771899 26142 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:26.772554 26142 heartbeater.cc:511] Master 127.25.16.124:34957 requested a full tablet report, sending...
I20240502 21:25:26.772826 26144 heartbeater.cc:347] Connected to a master server at 127.25.16.125:45259
I20240502 21:25:26.772974 26144 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:26.773509 26144 heartbeater.cc:511] Master 127.25.16.125:45259 requested a full tablet report, sending...
I20240502 21:25:26.774672 25825 ts_manager.cc:194] Registered new tserver with Master: 5efb33455c734d2599b359f95a194ffb (127.25.16.67:37545)
I20240502 21:25:26.774956 25759 ts_manager.cc:194] Registered new tserver with Master: 5efb33455c734d2599b359f95a194ffb (127.25.16.67:37545)
I20240502 21:25:26.777963 26143 heartbeater.cc:347] Connected to a master server at 127.25.16.126:33669
I20240502 21:25:26.778108 26143 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:26.778592 26143 heartbeater.cc:511] Master 127.25.16.126:33669 requested a full tablet report, sending...
I20240502 21:25:26.779407 25695 ts_manager.cc:194] Registered new tserver with Master: 5efb33455c734d2599b359f95a194ffb (127.25.16.67:37545)
I20240502 21:25:26.781149 25695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54180
I20240502 21:25:26.782061 25665 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01829339s
I20240502 21:25:27.383311 25894 catalog_manager.cc:1259] Loaded cluster ID: 5c2233d96d81481ea85062a9bd7d70fb
I20240502 21:25:27.383412 25894 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c: loading cluster ID for follower catalog manager: success
I20240502 21:25:27.385494 25894 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c: acquiring CA information for follower catalog manager: success
I20240502 21:25:27.386729 25894 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 21:25:27.388603 25903 catalog_manager.cc:1259] Loaded cluster ID: 5c2233d96d81481ea85062a9bd7d70fb
I20240502 21:25:27.388685 25903 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79: loading cluster ID for follower catalog manager: success
I20240502 21:25:27.390480 25903 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79: acquiring CA information for follower catalog manager: success
I20240502 21:25:27.391407 25903 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 21:25:27.481631 25695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54172
I20240502 21:25:27.568840 25695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54174
I20240502 21:25:27.784616 26143 heartbeater.cc:503] Master 127.25.16.126:33669 was elected leader, sending a full tablet report...
I20240502 21:25:28.371508 25889 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:28.482772 25976 heartbeater.cc:503] Master 127.25.16.126:33669 was elected leader, sending a full tablet report...
I20240502 21:25:28.569945 26054 heartbeater.cc:503] Master 127.25.16.126:33669 was elected leader, sending a full tablet report...
I20240502 21:25:28.836014 25665 test_util.cc:268] Using random seed: 1040286607
I20240502 21:25:28.875967 25695 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54182:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 21:25:28.919322 26018 tablet_service.cc:1467] Processing CreateTablet for tablet bd2fca5cf6bf4032915eb09dfb97ca5d (DEFAULT_TABLE table=test-workload [id=86a7b0dbb60b48acb509b571b8636d11]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 21:25:28.920183 26018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2fca5cf6bf4032915eb09dfb97ca5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:28.920374 26107 tablet_service.cc:1467] Processing CreateTablet for tablet f23b0ccf264f4d488febb1a95ee44599 (DEFAULT_TABLE table=test-workload [id=86a7b0dbb60b48acb509b571b8636d11]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 21:25:28.920894 26107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f23b0ccf264f4d488febb1a95ee44599. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:28.932251 26178 tablet_bootstrap.cc:492] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb: Bootstrap starting.
I20240502 21:25:28.934144 26177 tablet_bootstrap.cc:492] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad: Bootstrap starting.
I20240502 21:25:28.934337 25940 tablet_service.cc:1467] Processing CreateTablet for tablet 621fc6fad5fb4dfc9f342f82eeb03be5 (DEFAULT_TABLE table=test-workload [id=86a7b0dbb60b48acb509b571b8636d11]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 21:25:28.934368 25939 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc66d6103ba485993d796d103d612a8 (DEFAULT_TABLE table=test-workload [id=86a7b0dbb60b48acb509b571b8636d11]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 21:25:28.934957 25940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 621fc6fad5fb4dfc9f342f82eeb03be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:28.935185 25939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc66d6103ba485993d796d103d612a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:28.935559 26178 tablet_bootstrap.cc:654] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:28.936247 26177 tablet_bootstrap.cc:654] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:28.941473 26177 tablet_bootstrap.cc:492] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad: No bootstrap required, opened a new log
I20240502 21:25:28.941740 26177 ts_tablet_manager.cc:1367] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 21:25:28.942692 26177 raft_consensus.cc:348] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c29cdbaf98d424fb6cb46a5609d60ad" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35489 } }
I20240502 21:25:28.942910 26177 raft_consensus.cc:374] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:28.943308 26177 raft_consensus.cc:729] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c29cdbaf98d424fb6cb46a5609d60ad, State: Initialized, Role: FOLLOWER
I20240502 21:25:28.943667 26177 consensus_queue.cc:260] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c29cdbaf98d424fb6cb46a5609d60ad" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35489 } }
I20240502 21:25:28.944036 26177 raft_consensus.cc:388] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:28.944152 26177 raft_consensus.cc:482] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:28.944275 26177 raft_consensus.cc:3037] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:28.947278 26177 raft_consensus.cc:504] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c29cdbaf98d424fb6cb46a5609d60ad" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35489 } }
I20240502 21:25:28.947571 26177 leader_election.cc:302] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c29cdbaf98d424fb6cb46a5609d60ad; no voters:
I20240502 21:25:28.948304 26182 raft_consensus.cc:2781] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:28.948585 26177 leader_election.cc:288] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:28.949262 26178 tablet_bootstrap.cc:492] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb: No bootstrap required, opened a new log
I20240502 21:25:28.949494 26178 ts_tablet_manager.cc:1367] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.016s
I20240502 21:25:28.950277 26178 raft_consensus.cc:348] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efb33455c734d2599b359f95a194ffb" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 37545 } }
I20240502 21:25:28.950450 26178 raft_consensus.cc:374] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:28.950547 26178 raft_consensus.cc:729] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5efb33455c734d2599b359f95a194ffb, State: Initialized, Role: FOLLOWER
I20240502 21:25:28.950780 26178 consensus_queue.cc:260] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efb33455c734d2599b359f95a194ffb" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 37545 } }
I20240502 21:25:28.951035 26178 raft_consensus.cc:388] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:28.951153 26178 raft_consensus.cc:482] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:28.951295 26178 raft_consensus.cc:3037] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:28.952622 26177 ts_tablet_manager.cc:1398] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20240502 21:25:28.953536 26182 raft_consensus.cc:686] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [term 1 LEADER]: Becoming Leader. State: Replica: 7c29cdbaf98d424fb6cb46a5609d60ad, State: Running, Role: LEADER
I20240502 21:25:28.953920 26182 consensus_queue.cc:237] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c29cdbaf98d424fb6cb46a5609d60ad" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35489 } }
I20240502 21:25:28.954221 26178 raft_consensus.cc:504] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efb33455c734d2599b359f95a194ffb" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 37545 } }
I20240502 21:25:28.954442 26178 leader_election.cc:302] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5efb33455c734d2599b359f95a194ffb; no voters:
I20240502 21:25:28.955050 26181 tablet_bootstrap.cc:492] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8: Bootstrap starting.
I20240502 21:25:28.955799 26186 raft_consensus.cc:2781] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:28.957039 26181 tablet_bootstrap.cc:654] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:28.958040 26178 leader_election.cc:288] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:28.959203 26181 tablet_bootstrap.cc:492] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8: No bootstrap required, opened a new log
I20240502 21:25:28.959386 26181 ts_tablet_manager.cc:1367] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240502 21:25:28.960194 26181 raft_consensus.cc:348] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d303291ceac849de920a5e02566742c8" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39061 } }
I20240502 21:25:28.960458 26181 raft_consensus.cc:374] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:28.960666 26181 raft_consensus.cc:729] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d303291ceac849de920a5e02566742c8, State: Initialized, Role: FOLLOWER
I20240502 21:25:28.960994 26181 consensus_queue.cc:260] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d303291ceac849de920a5e02566742c8" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39061 } }
I20240502 21:25:28.961264 26181 raft_consensus.cc:388] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:28.960932 25695 catalog_manager.cc:5484] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c29cdbaf98d424fb6cb46a5609d60ad (127.25.16.66). New cstate: current_term: 1 leader_uuid: "7c29cdbaf98d424fb6cb46a5609d60ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c29cdbaf98d424fb6cb46a5609d60ad" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35489 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:28.961730 26178 ts_tablet_manager.cc:1398] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb: Time spent starting tablet: real 0.012s user 0.004s sys 0.002s
I20240502 21:25:28.963163 26181 raft_consensus.cc:482] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:28.963357 26181 raft_consensus.cc:3037] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:28.968601 26181 raft_consensus.cc:504] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d303291ceac849de920a5e02566742c8" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39061 } }
I20240502 21:25:28.968919 26181 leader_election.cc:302] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d303291ceac849de920a5e02566742c8; no voters:
I20240502 21:25:28.969727 26191 raft_consensus.cc:2781] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:28.969853 26186 raft_consensus.cc:686] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [term 1 LEADER]: Becoming Leader. State: Replica: 5efb33455c734d2599b359f95a194ffb, State: Running, Role: LEADER
I20240502 21:25:28.970033 26181 leader_election.cc:288] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:28.973004 26186 consensus_queue.cc:237] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efb33455c734d2599b359f95a194ffb" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 37545 } }
I20240502 21:25:28.975317 26181 ts_tablet_manager.cc:1398] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8: Time spent starting tablet: real 0.016s user 0.005s sys 0.004s
I20240502 21:25:28.975633 26181 tablet_bootstrap.cc:492] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8: Bootstrap starting.
I20240502 21:25:28.977141 26181 tablet_bootstrap.cc:654] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:28.980265 26181 tablet_bootstrap.cc:492] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8: No bootstrap required, opened a new log
I20240502 21:25:28.980464 26181 ts_tablet_manager.cc:1367] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 21:25:28.981349 26181 raft_consensus.cc:348] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d303291ceac849de920a5e02566742c8" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39061 } }
I20240502 21:25:28.981580 26181 raft_consensus.cc:374] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:28.981678 26181 raft_consensus.cc:729] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d303291ceac849de920a5e02566742c8, State: Initialized, Role: FOLLOWER
I20240502 21:25:28.981916 26181 consensus_queue.cc:260] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d303291ceac849de920a5e02566742c8" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39061 } }
I20240502 21:25:28.982100 26181 raft_consensus.cc:388] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:28.982203 26181 raft_consensus.cc:482] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:28.982336 26181 raft_consensus.cc:3037] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:28.983034 25695 catalog_manager.cc:5484] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5efb33455c734d2599b359f95a194ffb (127.25.16.67). New cstate: current_term: 1 leader_uuid: "5efb33455c734d2599b359f95a194ffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efb33455c734d2599b359f95a194ffb" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 37545 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:28.985564 26181 raft_consensus.cc:504] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d303291ceac849de920a5e02566742c8" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39061 } }
I20240502 21:25:28.985805 26181 leader_election.cc:302] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d303291ceac849de920a5e02566742c8; no voters:
I20240502 21:25:28.986068 26193 raft_consensus.cc:2781] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:28.986202 26193 raft_consensus.cc:686] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [term 1 LEADER]: Becoming Leader. State: Replica: d303291ceac849de920a5e02566742c8, State: Running, Role: LEADER
I20240502 21:25:28.986464 26193 consensus_queue.cc:237] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d303291ceac849de920a5e02566742c8" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39061 } }
I20240502 21:25:28.989903 26191 raft_consensus.cc:686] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [term 1 LEADER]: Becoming Leader. State: Replica: d303291ceac849de920a5e02566742c8, State: Running, Role: LEADER
I20240502 21:25:28.990280 26191 consensus_queue.cc:237] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d303291ceac849de920a5e02566742c8" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39061 } }
I20240502 21:25:28.990895 26181 leader_election.cc:288] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:28.991899 26181 ts_tablet_manager.cc:1398] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20240502 21:25:28.998766 25695 catalog_manager.cc:5484] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d303291ceac849de920a5e02566742c8 (127.25.16.65). New cstate: current_term: 1 leader_uuid: "d303291ceac849de920a5e02566742c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d303291ceac849de920a5e02566742c8" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39061 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:29.004731 25695 catalog_manager.cc:5484] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d303291ceac849de920a5e02566742c8 (127.25.16.65). New cstate: current_term: 1 leader_uuid: "d303291ceac849de920a5e02566742c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d303291ceac849de920a5e02566742c8" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39061 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:29.016223 25665 tablet_server.cc:178] TabletServer@127.25.16.65:0 shutting down...
I20240502 21:25:29.032474 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:25:29.032840 25665 tablet_replica.cc:330] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8: stopping tablet replica
I20240502 21:25:29.033221 25665 raft_consensus.cc:2229] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:29.033445 25665 raft_consensus.cc:2258] T 621fc6fad5fb4dfc9f342f82eeb03be5 P d303291ceac849de920a5e02566742c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:29.034798 25665 tablet_replica.cc:330] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8: stopping tablet replica
I20240502 21:25:29.034933 25665 raft_consensus.cc:2229] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:29.035069 25665 raft_consensus.cc:2258] T 6dc66d6103ba485993d796d103d612a8 P d303291ceac849de920a5e02566742c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:29.049928 25665 tablet_server.cc:194] TabletServer@127.25.16.65:0 shutdown complete.
I20240502 21:25:29.055020 25665 tablet_server.cc:178] TabletServer@127.25.16.66:0 shutting down...
I20240502 21:25:29.063364 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:25:29.063665 25665 tablet_replica.cc:330] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad: stopping tablet replica
I20240502 21:25:29.063815 25665 raft_consensus.cc:2229] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:29.063978 25665 raft_consensus.cc:2258] T bd2fca5cf6bf4032915eb09dfb97ca5d P 7c29cdbaf98d424fb6cb46a5609d60ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:29.077508 25665 tablet_server.cc:194] TabletServer@127.25.16.66:0 shutdown complete.
I20240502 21:25:29.079967 25665 tablet_server.cc:178] TabletServer@127.25.16.67:0 shutting down...
I20240502 21:25:29.087944 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:25:29.088338 25665 tablet_replica.cc:330] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb: stopping tablet replica
I20240502 21:25:29.088495 25665 raft_consensus.cc:2229] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:29.088647 25665 raft_consensus.cc:2258] T f23b0ccf264f4d488febb1a95ee44599 P 5efb33455c734d2599b359f95a194ffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:29.102241 25665 tablet_server.cc:194] TabletServer@127.25.16.67:0 shutdown complete.
I20240502 21:25:29.104825 25665 master.cc:537] Master@127.25.16.126:33669 shutting down...
I20240502 21:25:29.112179 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:29.112610 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:29.112763 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8984b42adf0a49e8ae78ec1755f26349: stopping tablet replica
I20240502 21:25:29.127059 25665 master.cc:552] Master@127.25.16.126:33669 shutdown complete.
I20240502 21:25:29.132000 25665 master.cc:537] Master@127.25.16.125:45259 shutting down...
I20240502 21:25:29.138527 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:29.138712 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:29.138810 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P a44dde6cefe642149ef3bb78cbc98d79: stopping tablet replica
I20240502 21:25:29.152101 25665 master.cc:552] Master@127.25.16.125:45259 shutdown complete.
I20240502 21:25:29.155944 25665 master.cc:537] Master@127.25.16.124:34957 shutting down...
I20240502 21:25:29.162073 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:29.162262 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:29.162379 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P b2d2c3d65b7c4b1e86edce8e6242fb9c: stopping tablet replica
I20240502 21:25:29.165652 25665 master.cc:552] Master@127.25.16.124:34957 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3068 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240502 21:25:29.178035 25665 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.16.126:32883
I20240502 21:25:29.178494 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:29.180692 26200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:29.180888 26201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:29.181326 26203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:29.181841 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:29.182058 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:29.182123 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:29.182163 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685129182148 us; error 0 us; skew 500 ppm
I20240502 21:25:29.182286 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:29.183378 25665 webserver.cc:460] Webserver started at http://127.25.16.126:37399/ using document root <none> and password file <none>
I20240502 21:25:29.183578 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:29.183636 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:29.183755 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:29.184387 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/master-0-root/instance:
uuid: "8bdd8e5ce50549cc8553aba7652a17c2"
format_stamp: "Formatted at 2024-05-02 21:25:29 on dist-test-slave-b524"
I20240502 21:25:29.186710 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:29.188169 26208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:29.188510 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:25:29.188612 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/master-0-root
uuid: "8bdd8e5ce50549cc8553aba7652a17c2"
format_stamp: "Formatted at 2024-05-02 21:25:29 on dist-test-slave-b524"
I20240502 21:25:29.188733 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:29.202853 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:29.203389 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:29.217134 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.126:32883
I20240502 21:25:29.217156 26259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.126:32883 every 8 connection(s)
I20240502 21:25:29.218464 26260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.222530 26260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2: Bootstrap starting.
I20240502 21:25:29.223731 26260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.225207 26260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2: No bootstrap required, opened a new log
I20240502 21:25:29.225806 26260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdd8e5ce50549cc8553aba7652a17c2" member_type: VOTER }
I20240502 21:25:29.225935 26260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:29.226022 26260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bdd8e5ce50549cc8553aba7652a17c2, State: Initialized, Role: FOLLOWER
I20240502 21:25:29.226213 26260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdd8e5ce50549cc8553aba7652a17c2" member_type: VOTER }
I20240502 21:25:29.226367 26260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:29.226438 26260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:29.226545 26260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:29.228648 26260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdd8e5ce50549cc8553aba7652a17c2" member_type: VOTER }
I20240502 21:25:29.228883 26260 leader_election.cc:302] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bdd8e5ce50549cc8553aba7652a17c2; no voters:
I20240502 21:25:29.229250 26260 leader_election.cc:288] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:29.229384 26263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:29.229866 26263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [term 1 LEADER]: Becoming Leader. State: Replica: 8bdd8e5ce50549cc8553aba7652a17c2, State: Running, Role: LEADER
I20240502 21:25:29.230047 26260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:29.230108 26263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdd8e5ce50549cc8553aba7652a17c2" member_type: VOTER }
I20240502 21:25:29.230968 26264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bdd8e5ce50549cc8553aba7652a17c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdd8e5ce50549cc8553aba7652a17c2" member_type: VOTER } }
I20240502 21:25:29.231019 26265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bdd8e5ce50549cc8553aba7652a17c2. Latest consensus state: current_term: 1 leader_uuid: "8bdd8e5ce50549cc8553aba7652a17c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdd8e5ce50549cc8553aba7652a17c2" member_type: VOTER } }
I20240502 21:25:29.231276 26265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:29.231266 26264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:29.235472 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:25:29.235473 26268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:29.236855 26268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:29.239392 26268 catalog_manager.cc:1347] Generated new cluster ID: 7d069a9717fd4eaab7f209a0a7028eff
I20240502 21:25:29.239477 26268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:29.246449 26268 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:29.247153 26268 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:29.254173 26268 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2: Generated new TSK 0
I20240502 21:25:29.254379 26268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:29.267971 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:29.270723 26281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:29.270884 26282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:29.271337 26284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:29.271608 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:29.271838 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:29.271899 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:29.271968 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685129271937 us; error 0 us; skew 500 ppm
I20240502 21:25:29.272106 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:29.273257 25665 webserver.cc:460] Webserver started at http://127.25.16.65:43529/ using document root <none> and password file <none>
I20240502 21:25:29.273473 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:29.273561 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:29.273692 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:29.274307 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-0-root/instance:
uuid: "7a8570ae144d43dd95f9d6bf521ba178"
format_stamp: "Formatted at 2024-05-02 21:25:29 on dist-test-slave-b524"
I20240502 21:25:29.276685 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:25:29.278152 26289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:29.278488 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 21:25:29.278589 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-0-root
uuid: "7a8570ae144d43dd95f9d6bf521ba178"
format_stamp: "Formatted at 2024-05-02 21:25:29 on dist-test-slave-b524"
I20240502 21:25:29.278726 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:29.318400 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:29.318997 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:29.319568 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:29.320487 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:29.320557 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:29.320678 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:29.320732 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:29.334765 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.65:33987
I20240502 21:25:29.334795 26351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.65:33987 every 8 connection(s)
I20240502 21:25:29.336328 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:29.339147 26356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:29.339278 26357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:29.341907 25665 server_base.cc:1034] running on GCE node
W20240502 21:25:29.342478 26359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:29.342866 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:29.342967 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:29.343015 26352 heartbeater.cc:347] Connected to a master server at 127.25.16.126:32883
I20240502 21:25:29.343031 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685129343001 us; error 0 us; skew 500 ppm
I20240502 21:25:29.343166 26352 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:29.343228 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:29.343518 26352 heartbeater.cc:511] Master 127.25.16.126:32883 requested a full tablet report, sending...
I20240502 21:25:29.344441 26225 ts_manager.cc:194] Registered new tserver with Master: 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987)
I20240502 21:25:29.344520 25665 webserver.cc:460] Webserver started at http://127.25.16.66:40947/ using document root <none> and password file <none>
I20240502 21:25:29.344741 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:29.344820 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:29.344975 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:29.345594 26225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50752
I20240502 21:25:29.345592 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-1-root/instance:
uuid: "4fc6cc31d05a40dcad5195a10cd0a485"
format_stamp: "Formatted at 2024-05-02 21:25:29 on dist-test-slave-b524"
I20240502 21:25:29.347895 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:29.349368 26364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:29.349681 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 21:25:29.349792 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-1-root
uuid: "4fc6cc31d05a40dcad5195a10cd0a485"
format_stamp: "Formatted at 2024-05-02 21:25:29 on dist-test-slave-b524"
I20240502 21:25:29.349905 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:29.361995 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:29.362474 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:29.362967 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:29.363682 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:29.363768 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:29.363852 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:29.363925 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:29.376355 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.66:37525
I20240502 21:25:29.376408 26426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.66:37525 every 8 connection(s)
I20240502 21:25:29.377918 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:29.380796 26430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:29.380864 26431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:29.382115 26433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:29.384120 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:29.384424 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:29.384514 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:29.384591 26427 heartbeater.cc:347] Connected to a master server at 127.25.16.126:32883
I20240502 21:25:29.384598 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685129384554 us; error 0 us; skew 500 ppm
I20240502 21:25:29.384733 26427 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:29.384796 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:29.385130 26427 heartbeater.cc:511] Master 127.25.16.126:32883 requested a full tablet report, sending...
I20240502 21:25:29.386025 25665 webserver.cc:460] Webserver started at http://127.25.16.67:35159/ using document root <none> and password file <none>
I20240502 21:25:29.386046 26225 ts_manager.cc:194] Registered new tserver with Master: 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:29.386216 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:29.386299 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:29.386399 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:29.386886 26225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50756
I20240502 21:25:29.387029 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-2-root/instance:
uuid: "ecce04fffd714216b8bebcff5ed2dc9e"
format_stamp: "Formatted at 2024-05-02 21:25:29 on dist-test-slave-b524"
I20240502 21:25:29.389565 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 21:25:29.390996 26438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:29.391306 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 21:25:29.391397 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-2-root
uuid: "ecce04fffd714216b8bebcff5ed2dc9e"
format_stamp: "Formatted at 2024-05-02 21:25:29 on dist-test-slave-b524"
I20240502 21:25:29.391505 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:29.410269 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:29.410750 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:29.411226 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:29.411955 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:29.412019 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:29.412093 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:29.412135 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:29.424937 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.67:35049
I20240502 21:25:29.424968 26500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.67:35049 every 8 connection(s)
I20240502 21:25:29.431037 26501 heartbeater.cc:347] Connected to a master server at 127.25.16.126:32883
I20240502 21:25:29.431164 26501 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:29.431519 26501 heartbeater.cc:511] Master 127.25.16.126:32883 requested a full tablet report, sending...
I20240502 21:25:29.432272 26225 ts_manager.cc:194] Registered new tserver with Master: ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67:35049)
I20240502 21:25:29.432587 25665 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006768777s
I20240502 21:25:29.433125 26225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50772
I20240502 21:25:30.346740 26352 heartbeater.cc:503] Master 127.25.16.126:32883 was elected leader, sending a full tablet report...
I20240502 21:25:30.388167 26427 heartbeater.cc:503] Master 127.25.16.126:32883 was elected leader, sending a full tablet report...
I20240502 21:25:30.434322 26501 heartbeater.cc:503] Master 127.25.16.126:32883 was elected leader, sending a full tablet report...
I20240502 21:25:30.459601 25665 test_util.cc:268] Using random seed: 1041910189
I20240502 21:25:30.470917 26225 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50778:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 21:25:30.471614 26225 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:25:30.503417 26392 tablet_service.cc:1467] Processing CreateTablet for tablet e9f57214b379402ca24433fc7a12df42 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 21:25:30.503661 26390 tablet_service.cc:1467] Processing CreateTablet for tablet c80c249257f442598b2d3b7555fda8f5 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 21:25:30.504029 26392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f57214b379402ca24433fc7a12df42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.504555 26388 tablet_service.cc:1467] Processing CreateTablet for tablet 8328927a71f0468fb874133b3df2ae42 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 21:25:30.504633 26386 tablet_service.cc:1467] Processing CreateTablet for tablet 04f17dd6a0804a39a6748b673d55c0d0 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 21:25:30.505091 26388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8328927a71f0468fb874133b3df2ae42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.505064 26387 tablet_service.cc:1467] Processing CreateTablet for tablet 7723674bc40a4df5a9dea66778d72236 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 21:25:30.504654 26389 tablet_service.cc:1467] Processing CreateTablet for tablet 1277d64e9fc9404aa08c009dba377adf (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 21:25:30.505520 26387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7723674bc40a4df5a9dea66778d72236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.505769 26389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1277d64e9fc9404aa08c009dba377adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.504207 26391 tablet_service.cc:1467] Processing CreateTablet for tablet 783f9c409f6543dfa54f7d764fd9b2db (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 21:25:30.506129 26391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783f9c409f6543dfa54f7d764fd9b2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.506326 26386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f17dd6a0804a39a6748b673d55c0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.508181 26390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c80c249257f442598b2d3b7555fda8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.511831 26317 tablet_service.cc:1467] Processing CreateTablet for tablet e9f57214b379402ca24433fc7a12df42 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 21:25:30.512007 26316 tablet_service.cc:1467] Processing CreateTablet for tablet 783f9c409f6543dfa54f7d764fd9b2db (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 21:25:30.512351 26317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f57214b379402ca24433fc7a12df42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.512537 26316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783f9c409f6543dfa54f7d764fd9b2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.513043 26315 tablet_service.cc:1467] Processing CreateTablet for tablet c80c249257f442598b2d3b7555fda8f5 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 21:25:30.513479 26315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c80c249257f442598b2d3b7555fda8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.516265 26310 tablet_service.cc:1467] Processing CreateTablet for tablet 8042e4ef446943f1be34424b94260767 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 21:25:30.516731 26310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8042e4ef446943f1be34424b94260767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.516697 26313 tablet_service.cc:1467] Processing CreateTablet for tablet 8328927a71f0468fb874133b3df2ae42 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 21:25:30.517160 26313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8328927a71f0468fb874133b3df2ae42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.520267 26464 tablet_service.cc:1467] Processing CreateTablet for tablet c80c249257f442598b2d3b7555fda8f5 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 21:25:30.520642 26312 tablet_service.cc:1467] Processing CreateTablet for tablet 7723674bc40a4df5a9dea66778d72236 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 21:25:30.520867 26464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c80c249257f442598b2d3b7555fda8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.520756 26311 tablet_service.cc:1467] Processing CreateTablet for tablet 04f17dd6a0804a39a6748b673d55c0d0 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 21:25:30.521132 26312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7723674bc40a4df5a9dea66778d72236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.521059 26466 tablet_service.cc:1467] Processing CreateTablet for tablet e9f57214b379402ca24433fc7a12df42 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 21:25:30.521526 26466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f57214b379402ca24433fc7a12df42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.521615 26462 tablet_service.cc:1467] Processing CreateTablet for tablet 8328927a71f0468fb874133b3df2ae42 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 21:25:30.522130 26462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8328927a71f0468fb874133b3df2ae42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.522892 26314 tablet_service.cc:1467] Processing CreateTablet for tablet 1277d64e9fc9404aa08c009dba377adf (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 21:25:30.523391 26314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1277d64e9fc9404aa08c009dba377adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.524720 26385 tablet_service.cc:1467] Processing CreateTablet for tablet 8042e4ef446943f1be34424b94260767 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 21:25:30.525260 26385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8042e4ef446943f1be34424b94260767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.528947 26463 tablet_service.cc:1467] Processing CreateTablet for tablet 1277d64e9fc9404aa08c009dba377adf (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 21:25:30.528837 26460 tablet_service.cc:1467] Processing CreateTablet for tablet 04f17dd6a0804a39a6748b673d55c0d0 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 21:25:30.530448 26463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1277d64e9fc9404aa08c009dba377adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.528728 26465 tablet_service.cc:1467] Processing CreateTablet for tablet 783f9c409f6543dfa54f7d764fd9b2db (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 21:25:30.530444 26311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f17dd6a0804a39a6748b673d55c0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.531145 26459 tablet_service.cc:1467] Processing CreateTablet for tablet 8042e4ef446943f1be34424b94260767 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 21:25:30.530452 26461 tablet_service.cc:1467] Processing CreateTablet for tablet 7723674bc40a4df5a9dea66778d72236 (DEFAULT_TABLE table=test-workload [id=e14070183f8348039a7d609649f0c01f]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 21:25:30.539166 26460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f17dd6a0804a39a6748b673d55c0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.542599 26465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783f9c409f6543dfa54f7d764fd9b2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.543223 26459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8042e4ef446943f1be34424b94260767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.547683 26461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7723674bc40a4df5a9dea66778d72236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.563733 26521 tablet_bootstrap.cc:492] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485: Bootstrap starting.
I20240502 21:25:30.568818 26523 tablet_bootstrap.cc:492] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e: Bootstrap starting.
I20240502 21:25:30.572955 26522 tablet_bootstrap.cc:492] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178: Bootstrap starting.
I20240502 21:25:30.577973 26521 tablet_bootstrap.cc:654] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.578660 26523 tablet_bootstrap.cc:654] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.579083 26522 tablet_bootstrap.cc:654] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.580585 26521 tablet_bootstrap.cc:492] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485: No bootstrap required, opened a new log
I20240502 21:25:30.580762 26521 ts_tablet_manager.cc:1367] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240502 21:25:30.581151 26522 tablet_bootstrap.cc:492] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178: No bootstrap required, opened a new log
I20240502 21:25:30.581290 26522 ts_tablet_manager.cc:1367] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 21:25:30.581692 26521 raft_consensus.cc:348] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } }
I20240502 21:25:30.581945 26521 raft_consensus.cc:374] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.581868 26522 raft_consensus.cc:348] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.582062 26521 raft_consensus.cc:729] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc6cc31d05a40dcad5195a10cd0a485, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.582067 26522 raft_consensus.cc:374] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.582206 26522 raft_consensus.cc:729] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8570ae144d43dd95f9d6bf521ba178, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.582326 26521 consensus_queue.cc:260] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } }
I20240502 21:25:30.582419 26522 consensus_queue.cc:260] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.583588 26522 ts_tablet_manager.cc:1398] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:30.583880 26522 tablet_bootstrap.cc:492] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178: Bootstrap starting.
I20240502 21:25:30.584138 26521 ts_tablet_manager.cc:1398] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:25:30.584460 26521 tablet_bootstrap.cc:492] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485: Bootstrap starting.
I20240502 21:25:30.585305 26522 tablet_bootstrap.cc:654] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.586098 26521 tablet_bootstrap.cc:654] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.588554 26523 tablet_bootstrap.cc:492] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e: No bootstrap required, opened a new log
I20240502 21:25:30.588722 26523 ts_tablet_manager.cc:1367] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.000s
I20240502 21:25:30.589510 26523 raft_consensus.cc:348] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.589712 26523 raft_consensus.cc:374] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.589812 26523 raft_consensus.cc:729] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.590055 26523 consensus_queue.cc:260] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.590818 26523 ts_tablet_manager.cc:1398] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:30.591135 26523 tablet_bootstrap.cc:492] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e: Bootstrap starting.
I20240502 21:25:30.593026 26523 tablet_bootstrap.cc:654] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.594401 26521 tablet_bootstrap.cc:492] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485: No bootstrap required, opened a new log
I20240502 21:25:30.594573 26521 ts_tablet_manager.cc:1367] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 21:25:30.595340 26521 raft_consensus.cc:348] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.595736 26522 tablet_bootstrap.cc:492] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178: No bootstrap required, opened a new log
I20240502 21:25:30.596115 26521 raft_consensus.cc:374] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.596144 26522 ts_tablet_manager.cc:1367] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20240502 21:25:30.596236 26521 raft_consensus.cc:729] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc6cc31d05a40dcad5195a10cd0a485, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.596707 26521 consensus_queue.cc:260] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.596818 26522 raft_consensus.cc:348] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.597023 26522 raft_consensus.cc:374] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.597110 26522 raft_consensus.cc:729] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8570ae144d43dd95f9d6bf521ba178, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.597368 26521 ts_tablet_manager.cc:1398] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:30.597373 26522 consensus_queue.cc:260] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.597666 26521 tablet_bootstrap.cc:492] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485: Bootstrap starting.
I20240502 21:25:30.597930 26522 ts_tablet_manager.cc:1398] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:30.599210 26521 tablet_bootstrap.cc:654] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.599866 26522 tablet_bootstrap.cc:492] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178: Bootstrap starting.
I20240502 21:25:30.601370 26522 tablet_bootstrap.cc:654] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.602562 26523 tablet_bootstrap.cc:492] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e: No bootstrap required, opened a new log
I20240502 21:25:30.602738 26523 ts_tablet_manager.cc:1367] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240502 21:25:30.603488 26523 raft_consensus.cc:348] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } }
I20240502 21:25:30.603734 26523 raft_consensus.cc:374] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.603818 26523 raft_consensus.cc:729] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.604070 26523 consensus_queue.cc:260] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } }
I20240502 21:25:30.604600 26523 ts_tablet_manager.cc:1398] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:30.604899 26523 tablet_bootstrap.cc:492] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e: Bootstrap starting.
I20240502 21:25:30.606712 26523 tablet_bootstrap.cc:654] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.609316 26522 tablet_bootstrap.cc:492] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178: No bootstrap required, opened a new log
I20240502 21:25:30.609493 26523 tablet_bootstrap.cc:492] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e: No bootstrap required, opened a new log
I20240502 21:25:30.609529 26522 ts_tablet_manager.cc:1367] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20240502 21:25:30.609526 26521 tablet_bootstrap.cc:492] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485: No bootstrap required, opened a new log
I20240502 21:25:30.609663 26523 ts_tablet_manager.cc:1367] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:25:30.609788 26521 ts_tablet_manager.cc:1367] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20240502 21:25:30.610306 26523 raft_consensus.cc:348] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.610337 26522 raft_consensus.cc:348] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.610474 26523 raft_consensus.cc:374] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.610536 26522 raft_consensus.cc:374] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.610599 26523 raft_consensus.cc:729] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.610652 26522 raft_consensus.cc:729] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8570ae144d43dd95f9d6bf521ba178, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.610471 26521 raft_consensus.cc:348] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.610814 26521 raft_consensus.cc:374] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.610921 26521 raft_consensus.cc:729] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc6cc31d05a40dcad5195a10cd0a485, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.610846 26523 consensus_queue.cc:260] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.610970 26522 consensus_queue.cc:260] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.611337 26523 ts_tablet_manager.cc:1398] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:30.611598 26523 tablet_bootstrap.cc:492] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e: Bootstrap starting.
I20240502 21:25:30.612375 26522 ts_tablet_manager.cc:1398] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:25:30.612682 26522 tablet_bootstrap.cc:492] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178: Bootstrap starting.
I20240502 21:25:30.611164 26521 consensus_queue.cc:260] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.613206 26523 tablet_bootstrap.cc:654] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.615293 26521 ts_tablet_manager.cc:1398] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:25:30.615639 26522 tablet_bootstrap.cc:654] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.616098 26521 tablet_bootstrap.cc:492] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485: Bootstrap starting.
I20240502 21:25:30.617950 26523 tablet_bootstrap.cc:492] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e: No bootstrap required, opened a new log
I20240502 21:25:30.617995 26522 tablet_bootstrap.cc:492] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178: No bootstrap required, opened a new log
I20240502 21:25:30.618145 26523 ts_tablet_manager.cc:1367] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 21:25:30.618167 26522 ts_tablet_manager.cc:1367] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:25:30.618297 26521 tablet_bootstrap.cc:654] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.618749 26522 raft_consensus.cc:348] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.618911 26522 raft_consensus.cc:374] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.618974 26522 raft_consensus.cc:729] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8570ae144d43dd95f9d6bf521ba178, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.618947 26523 raft_consensus.cc:348] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.619167 26523 raft_consensus.cc:374] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.619256 26523 raft_consensus.cc:729] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.619189 26522 consensus_queue.cc:260] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.619550 26523 consensus_queue.cc:260] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.619705 26522 ts_tablet_manager.cc:1398] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:25:30.619988 26522 tablet_bootstrap.cc:492] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178: Bootstrap starting.
I20240502 21:25:30.620048 26523 ts_tablet_manager.cc:1398] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:25:30.620966 26523 tablet_bootstrap.cc:492] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e: Bootstrap starting.
I20240502 21:25:30.621513 26522 tablet_bootstrap.cc:654] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.622643 26523 tablet_bootstrap.cc:654] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.627759 26522 tablet_bootstrap.cc:492] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178: No bootstrap required, opened a new log
I20240502 21:25:30.627876 26522 ts_tablet_manager.cc:1367] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240502 21:25:30.628188 26521 tablet_bootstrap.cc:492] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485: No bootstrap required, opened a new log
I20240502 21:25:30.628382 26521 ts_tablet_manager.cc:1367] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240502 21:25:30.628646 26523 tablet_bootstrap.cc:492] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e: No bootstrap required, opened a new log
I20240502 21:25:30.628533 26522 raft_consensus.cc:348] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.628741 26531 raft_consensus.cc:482] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.628827 26522 raft_consensus.cc:374] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.628933 26522 raft_consensus.cc:729] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8570ae144d43dd95f9d6bf521ba178, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.628893 26531 raft_consensus.cc:504] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } }
I20240502 21:25:30.628971 26523 ts_tablet_manager.cc:1367] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20240502 21:25:30.629077 26521 raft_consensus.cc:348] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.629169 26522 consensus_queue.cc:260] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.629334 26521 raft_consensus.cc:374] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.629431 26521 raft_consensus.cc:729] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc6cc31d05a40dcad5195a10cd0a485, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.629789 26522 ts_tablet_manager.cc:1398] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:30.629758 26521 consensus_queue.cc:260] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.630012 26531 leader_election.cc:288] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:30.630153 26522 tablet_bootstrap.cc:492] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178: Bootstrap starting.
I20240502 21:25:30.630236 26521 ts_tablet_manager.cc:1398] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:30.630450 26523 raft_consensus.cc:348] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.630626 26521 tablet_bootstrap.cc:492] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485: Bootstrap starting.
I20240502 21:25:30.630764 26523 raft_consensus.cc:374] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.630918 26523 raft_consensus.cc:729] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.631212 26523 consensus_queue.cc:260] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.632258 26521 tablet_bootstrap.cc:654] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.632400 26522 tablet_bootstrap.cc:654] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.636463 26523 ts_tablet_manager.cc:1398] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:30.636803 26523 tablet_bootstrap.cc:492] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e: Bootstrap starting.
I20240502 21:25:30.638530 26523 tablet_bootstrap.cc:654] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.643210 26521 tablet_bootstrap.cc:492] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485: No bootstrap required, opened a new log
I20240502 21:25:30.643354 26521 ts_tablet_manager.cc:1367] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20240502 21:25:30.644150 26521 raft_consensus.cc:348] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.644385 26521 raft_consensus.cc:374] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.644420 26528 raft_consensus.cc:482] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.644482 26521 raft_consensus.cc:729] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc6cc31d05a40dcad5195a10cd0a485, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.644575 26528 raft_consensus.cc:504] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.644742 26521 consensus_queue.cc:260] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.645694 26528 leader_election.cc:288] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525), ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67:35049)
I20240502 21:25:30.646204 26521 ts_tablet_manager.cc:1398] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:30.646473 26521 tablet_bootstrap.cc:492] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485: Bootstrap starting.
I20240502 21:25:30.647991 26521 tablet_bootstrap.cc:654] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.651604 26528 raft_consensus.cc:482] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.651780 26528 raft_consensus.cc:504] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.652432 26528 leader_election.cc:288] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525), ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67:35049)
I20240502 21:25:30.652820 26523 tablet_bootstrap.cc:492] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e: No bootstrap required, opened a new log
I20240502 21:25:30.652997 26523 ts_tablet_manager.cc:1367] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.003s
I20240502 21:25:30.653121 26522 tablet_bootstrap.cc:492] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178: No bootstrap required, opened a new log
I20240502 21:25:30.653484 26522 ts_tablet_manager.cc:1367] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.012s
I20240502 21:25:30.653698 26523 raft_consensus.cc:348] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.653951 26523 raft_consensus.cc:374] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.654035 26523 raft_consensus.cc:729] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.654219 26522 raft_consensus.cc:348] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.654302 26523 consensus_queue.cc:260] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.654448 26522 raft_consensus.cc:374] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.654529 26522 raft_consensus.cc:729] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8570ae144d43dd95f9d6bf521ba178, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.654958 26523 ts_tablet_manager.cc:1398] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:30.655356 26522 consensus_queue.cc:260] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.655920 26523 tablet_bootstrap.cc:492] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e: Bootstrap starting.
I20240502 21:25:30.657398 26521 tablet_bootstrap.cc:492] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485: No bootstrap required, opened a new log
I20240502 21:25:30.657617 26521 ts_tablet_manager.cc:1367] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20240502 21:25:30.658447 26522 ts_tablet_manager.cc:1398] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:25:30.658370 26521 raft_consensus.cc:348] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.658708 26521 raft_consensus.cc:374] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.658767 26522 tablet_bootstrap.cc:492] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178: Bootstrap starting.
I20240502 21:25:30.658903 26521 raft_consensus.cc:729] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc6cc31d05a40dcad5195a10cd0a485, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.659338 26521 consensus_queue.cc:260] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.660005 26521 ts_tablet_manager.cc:1398] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:30.660354 26521 tablet_bootstrap.cc:492] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485: Bootstrap starting.
I20240502 21:25:30.660729 26522 tablet_bootstrap.cc:654] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.658345 26523 tablet_bootstrap.cc:654] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.662709 26521 tablet_bootstrap.cc:654] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.665647 26402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f57214b379402ca24433fc7a12df42" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" is_pre_election: true
I20240502 21:25:30.665997 26402 raft_consensus.cc:2454] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 0.
I20240502 21:25:30.666261 26522 tablet_bootstrap.cc:492] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178: No bootstrap required, opened a new log
I20240502 21:25:30.666409 26522 ts_tablet_manager.cc:1367] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 21:25:30.666637 26441 leader_election.cc:302] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.666997 26531 raft_consensus.cc:2781] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.667184 26531 raft_consensus.cc:482] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.667302 26531 raft_consensus.cc:3037] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.667637 26527 raft_consensus.cc:482] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.667790 26527 raft_consensus.cc:504] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.668896 26527 leader_election.cc:288] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67:35049)
I20240502 21:25:30.670590 26531 raft_consensus.cc:504] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } }
I20240502 21:25:30.671178 26531 leader_election.cc:288] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Requested vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:30.671371 26523 tablet_bootstrap.cc:492] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e: No bootstrap required, opened a new log
I20240502 21:25:30.671545 26523 ts_tablet_manager.cc:1367] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20240502 21:25:30.672355 26523 raft_consensus.cc:348] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.672621 26523 raft_consensus.cc:374] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.672758 26523 raft_consensus.cc:729] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.673059 26523 consensus_queue.cc:260] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.675745 26523 ts_tablet_manager.cc:1398] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:25:30.678061 26523 tablet_bootstrap.cc:492] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e: Bootstrap starting.
I20240502 21:25:30.679714 26523 tablet_bootstrap.cc:654] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.681142 26402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f57214b379402ca24433fc7a12df42" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485"
I20240502 21:25:30.680054 26522 raft_consensus.cc:348] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.681349 26402 raft_consensus.cc:3037] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.681525 26522 raft_consensus.cc:374] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.681695 26522 raft_consensus.cc:729] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8570ae144d43dd95f9d6bf521ba178, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.681972 26522 consensus_queue.cc:260] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.682628 26522 ts_tablet_manager.cc:1398] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 21:25:30.682984 26522 tablet_bootstrap.cc:492] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178: Bootstrap starting.
I20240502 21:25:30.684629 26522 tablet_bootstrap.cc:654] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.682992 26521 tablet_bootstrap.cc:492] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485: No bootstrap required, opened a new log
I20240502 21:25:30.685788 26521 ts_tablet_manager.cc:1367] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.001s
I20240502 21:25:30.686530 26402 raft_consensus.cc:2454] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 1.
I20240502 21:25:30.686583 26521 raft_consensus.cc:348] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.686897 26521 raft_consensus.cc:374] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.686991 26441 leader_election.cc:302] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.687043 26521 raft_consensus.cc:729] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc6cc31d05a40dcad5195a10cd0a485, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.687264 26531 raft_consensus.cc:2781] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.687382 26521 consensus_queue.cc:260] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.688367 26521 ts_tablet_manager.cc:1398] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:30.688731 26521 tablet_bootstrap.cc:492] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485: Bootstrap starting.
I20240502 21:25:30.698326 26521 tablet_bootstrap.cc:654] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.699259 26327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8328927a71f0468fb874133b3df2ae42" candidate_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178" is_pre_election: true
I20240502 21:25:30.701457 26327 raft_consensus.cc:2454] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fc6cc31d05a40dcad5195a10cd0a485 in term 0.
I20240502 21:25:30.702080 26531 raft_consensus.cc:686] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Becoming Leader. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Running, Role: LEADER
I20240502 21:25:30.702405 26531 consensus_queue.cc:237] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } }
I20240502 21:25:30.702083 26367 leader_election.cc:302] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, 7a8570ae144d43dd95f9d6bf521ba178; no voters:
I20240502 21:25:30.704056 26557 raft_consensus.cc:482] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.704213 26557 raft_consensus.cc:504] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.705211 26557 leader_election.cc:288] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:30.705312 26327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f57214b379402ca24433fc7a12df42" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178" is_pre_election: true
I20240502 21:25:30.705463 26527 raft_consensus.cc:2781] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.705674 26527 raft_consensus.cc:482] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.705643 26402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7723674bc40a4df5a9dea66778d72236" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" is_pre_election: true
I20240502 21:25:30.705379 26557 raft_consensus.cc:482] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.706655 26527 raft_consensus.cc:3037] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.707496 26557 raft_consensus.cc:504] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.708086 26557 leader_election.cc:288] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
W20240502 21:25:30.708250 26441 leader_election.cc:341] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:30.708815 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f57214b379402ca24433fc7a12df42" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
I20240502 21:25:30.709631 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f17dd6a0804a39a6748b673d55c0d0" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" is_pre_election: true
I20240502 21:25:30.709810 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f17dd6a0804a39a6748b673d55c0d0" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178" is_pre_election: true
I20240502 21:25:30.709931 26401 raft_consensus.cc:2454] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 0.
I20240502 21:25:30.710021 26326 raft_consensus.cc:2454] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 0.
I20240502 21:25:30.710394 26441 leader_election.cc:302] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
W20240502 21:25:30.710712 26441 leader_election.cc:341] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:30.711174 26557 raft_consensus.cc:2781] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.711277 26557 raft_consensus.cc:482] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.711387 26557 raft_consensus.cc:3037] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.714582 26557 raft_consensus.cc:504] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.715262 26557 leader_election.cc:288] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Requested vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:30.715854 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f17dd6a0804a39a6748b673d55c0d0" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
I20240502 21:25:30.715901 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f17dd6a0804a39a6748b673d55c0d0" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485"
I20240502 21:25:30.716058 26326 raft_consensus.cc:3037] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.716099 26401 raft_consensus.cc:3037] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.719311 26326 raft_consensus.cc:2454] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 1.
I20240502 21:25:30.719450 26401 raft_consensus.cc:2454] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 1.
I20240502 21:25:30.719782 26441 leader_election.cc:302] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8570ae144d43dd95f9d6bf521ba178, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.721890 26402 raft_consensus.cc:2454] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 0.
I20240502 21:25:30.722375 26327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7723674bc40a4df5a9dea66778d72236" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178" is_pre_election: true
I20240502 21:25:30.722623 26557 raft_consensus.cc:2781] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.722668 26441 leader_election.cc:302] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.724495 26327 raft_consensus.cc:2454] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 0.
I20240502 21:25:30.725175 26531 raft_consensus.cc:2781] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.730579 26557 raft_consensus.cc:686] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Becoming Leader. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Running, Role: LEADER
I20240502 21:25:30.731292 26523 tablet_bootstrap.cc:492] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e: No bootstrap required, opened a new log
I20240502 21:25:30.734862 26531 raft_consensus.cc:482] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.735661 26523 ts_tablet_manager.cc:1367] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.013s
I20240502 21:25:30.735958 26531 raft_consensus.cc:3037] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.735198 26557 consensus_queue.cc:237] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.735232 26224 catalog_manager.cc:5484] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67). New cstate: current_term: 1 leader_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:30.744470 26527 raft_consensus.cc:504] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.745199 26527 leader_election.cc:288] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67:35049)
I20240502 21:25:30.746866 26327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8328927a71f0468fb874133b3df2ae42" candidate_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
I20240502 21:25:30.747496 26327 raft_consensus.cc:3037] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.748554 26522 tablet_bootstrap.cc:492] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178: No bootstrap required, opened a new log
I20240502 21:25:30.748703 26523 raft_consensus.cc:348] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.750545 26523 raft_consensus.cc:374] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.750689 26523 raft_consensus.cc:729] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.750936 26523 consensus_queue.cc:260] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.751477 26521 tablet_bootstrap.cc:492] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485: No bootstrap required, opened a new log
I20240502 21:25:30.751549 26523 ts_tablet_manager.cc:1398] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240502 21:25:30.751765 26521 ts_tablet_manager.cc:1367] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent bootstrapping tablet: real 0.063s user 0.013s sys 0.005s
I20240502 21:25:30.752586 26327 raft_consensus.cc:2454] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fc6cc31d05a40dcad5195a10cd0a485 in term 1.
I20240502 21:25:30.752599 26521 raft_consensus.cc:348] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.752985 26521 raft_consensus.cc:374] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.753012 26367 leader_election.cc:302] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, 7a8570ae144d43dd95f9d6bf521ba178; no voters:
I20240502 21:25:30.753072 26521 raft_consensus.cc:729] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc6cc31d05a40dcad5195a10cd0a485, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.753404 26527 raft_consensus.cc:2781] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.753391 26521 consensus_queue.cc:260] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.754724 26531 raft_consensus.cc:504] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.755776 26327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7723674bc40a4df5a9dea66778d72236" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
I20240502 21:25:30.755996 26327 raft_consensus.cc:3037] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.758996 26327 raft_consensus.cc:2454] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 1.
I20240502 21:25:30.759389 26441 leader_election.cc:302] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8570ae144d43dd95f9d6bf521ba178, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.759721 26558 raft_consensus.cc:2781] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.759872 26558 raft_consensus.cc:686] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Becoming Leader. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Running, Role: LEADER
I20240502 21:25:30.760229 26558 consensus_queue.cc:237] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.760810 26521 ts_tablet_manager.cc:1398] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240502 21:25:30.762950 26522 ts_tablet_manager.cc:1367] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178: Time spent bootstrapping tablet: real 0.066s user 0.017s sys 0.005s
I20240502 21:25:30.763108 26476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1277d64e9fc9404aa08c009dba377adf" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" is_pre_election: true
I20240502 21:25:30.763444 26476 raft_consensus.cc:2454] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8570ae144d43dd95f9d6bf521ba178 in term 0.
I20240502 21:25:30.764510 26293 leader_election.cc:302] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8570ae144d43dd95f9d6bf521ba178, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.764670 26531 leader_election.cc:288] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Requested vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:30.764751 26402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7723674bc40a4df5a9dea66778d72236" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485"
I20240502 21:25:30.765085 26528 raft_consensus.cc:2781] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.764864 26522 raft_consensus.cc:348] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } }
I20240502 21:25:30.765697 26558 raft_consensus.cc:482] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.765944 26402 raft_consensus.cc:3037] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.766901 26558 raft_consensus.cc:504] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.766355 26528 raft_consensus.cc:482] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.767654 26528 raft_consensus.cc:3037] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.768067 26476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8328927a71f0468fb874133b3df2ae42" candidate_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" is_pre_election: true
I20240502 21:25:30.768272 26476 raft_consensus.cc:2454] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fc6cc31d05a40dcad5195a10cd0a485 in term 0.
I20240502 21:25:30.765882 26475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8328927a71f0468fb874133b3df2ae42" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" is_pre_election: true
I20240502 21:25:30.768582 26475 raft_consensus.cc:2454] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8570ae144d43dd95f9d6bf521ba178 in term 0.
I20240502 21:25:30.769156 26522 raft_consensus.cc:374] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.768210 26474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8328927a71f0468fb874133b3df2ae42" candidate_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e"
I20240502 21:25:30.769724 26293 leader_election.cc:302] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8570ae144d43dd95f9d6bf521ba178, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.769848 26474 raft_consensus.cc:3037] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.770221 26522 raft_consensus.cc:729] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8570ae144d43dd95f9d6bf521ba178, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.770082 26224 catalog_manager.cc:5484] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67). New cstate: current_term: 1 leader_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:30.771605 26527 raft_consensus.cc:686] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 LEADER]: Becoming Leader. State: Replica: 4fc6cc31d05a40dcad5195a10cd0a485, State: Running, Role: LEADER
I20240502 21:25:30.771018 26522 consensus_queue.cc:260] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } }
I20240502 21:25:30.771672 26558 leader_election.cc:288] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:30.774492 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1277d64e9fc9404aa08c009dba377adf" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" is_pre_election: true
I20240502 21:25:30.778934 26401 raft_consensus.cc:2454] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 0.
I20240502 21:25:30.773886 26527 consensus_queue.cc:237] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.779716 26441 leader_election.cc:302] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.782784 26402 raft_consensus.cc:2454] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 1.
I20240502 21:25:30.784193 26558 raft_consensus.cc:2781] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.784340 26558 raft_consensus.cc:482] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.785502 26528 raft_consensus.cc:504] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.788249 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8328927a71f0468fb874133b3df2ae42" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" is_pre_election: true
I20240502 21:25:30.788846 26558 raft_consensus.cc:3037] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.791918 26558 raft_consensus.cc:504] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.792398 26528 leader_election.cc:288] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 1 election: Requested vote from peers 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525), ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67:35049)
I20240502 21:25:30.792757 26224 catalog_manager.cc:5484] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67). New cstate: current_term: 1 leader_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:30.793217 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1277d64e9fc9404aa08c009dba377adf" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485"
I20240502 21:25:30.793406 26401 raft_consensus.cc:3037] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.794740 26402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1277d64e9fc9404aa08c009dba377adf" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" is_pre_election: true
I20240502 21:25:30.794951 26474 raft_consensus.cc:2454] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fc6cc31d05a40dcad5195a10cd0a485 in term 1.
I20240502 21:25:30.796728 26522 ts_tablet_manager.cc:1398] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178: Time spent starting tablet: real 0.034s user 0.016s sys 0.008s
I20240502 21:25:30.796913 26400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1277d64e9fc9404aa08c009dba377adf" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485"
I20240502 21:25:30.792626 26528 raft_consensus.cc:2741] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240502 21:25:30.799847 26401 raft_consensus.cc:2454] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 1.
I20240502 21:25:30.800220 26441 leader_election.cc:302] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.792605 26558 leader_election.cc:288] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Requested vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:30.800534 26557 raft_consensus.cc:2781] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.800686 26557 raft_consensus.cc:686] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Becoming Leader. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Running, Role: LEADER
I20240502 21:25:30.801051 26557 consensus_queue.cc:237] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.801398 26474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1277d64e9fc9404aa08c009dba377adf" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e"
I20240502 21:25:30.802466 26327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1277d64e9fc9404aa08c009dba377adf" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178" is_pre_election: true
I20240502 21:25:30.802650 26293 leader_election.cc:302] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8570ae144d43dd95f9d6bf521ba178; no voters: 4fc6cc31d05a40dcad5195a10cd0a485, ecce04fffd714216b8bebcff5ed2dc9e
I20240502 21:25:30.802815 26327 raft_consensus.cc:2379] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecce04fffd714216b8bebcff5ed2dc9e in current term 1: Already voted for candidate 7a8570ae144d43dd95f9d6bf521ba178 in this term.
I20240502 21:25:30.803090 26566 raft_consensus.cc:2726] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:25:30.803884 26224 catalog_manager.cc:5484] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67). New cstate: current_term: 1 leader_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:30.804195 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1277d64e9fc9404aa08c009dba377adf" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
I20240502 21:25:30.804420 26326 raft_consensus.cc:2379] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ecce04fffd714216b8bebcff5ed2dc9e in current term 1: Already voted for candidate 7a8570ae144d43dd95f9d6bf521ba178 in this term.
I20240502 21:25:30.807325 26225 catalog_manager.cc:5484] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66). New cstate: current_term: 1 leader_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:30.807746 26557 raft_consensus.cc:482] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.807907 26557 raft_consensus.cc:504] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.808545 26557 leader_election.cc:288] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:30.808981 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8042e4ef446943f1be34424b94260767" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178" is_pre_election: true
I20240502 21:25:30.809078 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8042e4ef446943f1be34424b94260767" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" is_pre_election: true
I20240502 21:25:30.809171 26326 raft_consensus.cc:2454] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 0.
I20240502 21:25:30.809266 26401 raft_consensus.cc:2454] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 0.
I20240502 21:25:30.809523 26441 leader_election.cc:302] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8570ae144d43dd95f9d6bf521ba178, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.809820 26557 raft_consensus.cc:2781] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.809935 26557 raft_consensus.cc:482] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.810037 26557 raft_consensus.cc:3037] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.811702 26557 raft_consensus.cc:504] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.812146 26557 leader_election.cc:288] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Requested vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:30.812590 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8042e4ef446943f1be34424b94260767" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485"
I20240502 21:25:30.812662 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8042e4ef446943f1be34424b94260767" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
I20240502 21:25:30.812842 26326 raft_consensus.cc:3037] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.812839 26401 raft_consensus.cc:3037] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.815165 26401 raft_consensus.cc:2454] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 1.
I20240502 21:25:30.815287 26326 raft_consensus.cc:2454] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 1.
I20240502 21:25:30.815558 26441 leader_election.cc:302] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.815797 26557 raft_consensus.cc:2781] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.815954 26557 raft_consensus.cc:686] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Becoming Leader. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Running, Role: LEADER
I20240502 21:25:30.816164 26557 consensus_queue.cc:237] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.818271 26225 catalog_manager.cc:5484] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67). New cstate: current_term: 1 leader_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:30.839025 26557 raft_consensus.cc:482] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.839160 26557 raft_consensus.cc:504] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.839624 26557 leader_election.cc:288] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:30.839982 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80c249257f442598b2d3b7555fda8f5" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178" is_pre_election: true
I20240502 21:25:30.840032 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80c249257f442598b2d3b7555fda8f5" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" is_pre_election: true
I20240502 21:25:30.840170 26326 raft_consensus.cc:2454] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 0.
I20240502 21:25:30.840198 26401 raft_consensus.cc:2454] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 0.
I20240502 21:25:30.840505 26441 leader_election.cc:302] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8570ae144d43dd95f9d6bf521ba178, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.840720 26557 raft_consensus.cc:2781] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.840817 26557 raft_consensus.cc:482] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.840898 26557 raft_consensus.cc:3037] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.842646 26557 raft_consensus.cc:504] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.843104 26557 leader_election.cc:288] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Requested vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:30.843430 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80c249257f442598b2d3b7555fda8f5" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
I20240502 21:25:30.843515 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80c249257f442598b2d3b7555fda8f5" candidate_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485"
I20240502 21:25:30.843592 26326 raft_consensus.cc:3037] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.843688 26401 raft_consensus.cc:3037] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.845577 26326 raft_consensus.cc:2454] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 1.
I20240502 21:25:30.845641 26401 raft_consensus.cc:2454] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce04fffd714216b8bebcff5ed2dc9e in term 1.
I20240502 21:25:30.845897 26441 leader_election.cc:302] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8570ae144d43dd95f9d6bf521ba178, ecce04fffd714216b8bebcff5ed2dc9e; no voters:
I20240502 21:25:30.846139 26557 raft_consensus.cc:2781] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.846307 26557 raft_consensus.cc:686] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Becoming Leader. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Running, Role: LEADER
I20240502 21:25:30.846524 26557 consensus_queue.cc:237] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.848834 26225 catalog_manager.cc:5484] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e reported cstate change: term changed from 0 to 1, leader changed from <none> to ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67). New cstate: current_term: 1 leader_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:30.909471 26527 raft_consensus.cc:482] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.909674 26527 raft_consensus.cc:504] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.910259 26527 leader_election.cc:288] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67:35049)
I20240502 21:25:30.910698 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783f9c409f6543dfa54f7d764fd9b2db" candidate_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178" is_pre_election: true
I20240502 21:25:30.910723 26474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783f9c409f6543dfa54f7d764fd9b2db" candidate_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" is_pre_election: true
I20240502 21:25:30.911103 26474 raft_consensus.cc:2454] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fc6cc31d05a40dcad5195a10cd0a485 in term 0.
I20240502 21:25:30.911073 26326 raft_consensus.cc:2454] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fc6cc31d05a40dcad5195a10cd0a485 in term 0.
I20240502 21:25:30.911538 26367 leader_election.cc:302] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, 7a8570ae144d43dd95f9d6bf521ba178; no voters:
I20240502 21:25:30.911860 26527 raft_consensus.cc:2781] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.911950 26527 raft_consensus.cc:482] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.912029 26527 raft_consensus.cc:3037] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.913902 26527 raft_consensus.cc:504] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.914415 26527 leader_election.cc:288] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65:33987), ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67:35049)
I20240502 21:25:30.914822 26474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783f9c409f6543dfa54f7d764fd9b2db" candidate_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e"
I20240502 21:25:30.914842 26326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783f9c409f6543dfa54f7d764fd9b2db" candidate_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
I20240502 21:25:30.914996 26474 raft_consensus.cc:3037] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.915019 26326 raft_consensus.cc:3037] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.916992 26474 raft_consensus.cc:2454] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fc6cc31d05a40dcad5195a10cd0a485 in term 1.
I20240502 21:25:30.917055 26326 raft_consensus.cc:2454] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fc6cc31d05a40dcad5195a10cd0a485 in term 1.
I20240502 21:25:30.917408 26367 leader_election.cc:302] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, 7a8570ae144d43dd95f9d6bf521ba178; no voters:
I20240502 21:25:30.917716 26527 raft_consensus.cc:2781] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.917862 26527 raft_consensus.cc:686] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 LEADER]: Becoming Leader. State: Replica: 4fc6cc31d05a40dcad5195a10cd0a485, State: Running, Role: LEADER
I20240502 21:25:30.918097 26527 consensus_queue.cc:237] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:30.920619 26225 catalog_manager.cc:5484] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66). New cstate: current_term: 1 leader_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:30.997341 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:30.999934 26573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:31.000501 26574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:31.000957 26576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:31.001274 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:31.001504 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:31.001581 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:31.001637 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685131001619 us; error 0 us; skew 500 ppm
I20240502 21:25:31.001798 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:31.003197 25665 webserver.cc:460] Webserver started at http://127.25.16.68:33599/ using document root <none> and password file <none>
I20240502 21:25:31.003461 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:31.003545 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:31.003693 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:31.004405 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-3-root/instance:
uuid: "2cb9a05822ba4213a35821b97abdaf06"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-b524"
I20240502 21:25:31.006934 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:31.008533 26581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.008937 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 21:25:31.009061 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-3-root
uuid: "2cb9a05822ba4213a35821b97abdaf06"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-b524"
I20240502 21:25:31.009168 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714685126025081-25665-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:31.017263 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:31.017788 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:31.018308 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:31.019249 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:31.019361 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.019469 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:31.019558 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.035337 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.68:38255
I20240502 21:25:31.035380 26643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.68:38255 every 8 connection(s)
I20240502 21:25:31.041924 26644 heartbeater.cc:347] Connected to a master server at 127.25.16.126:32883
I20240502 21:25:31.042052 26644 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:31.042480 26644 heartbeater.cc:511] Master 127.25.16.126:32883 requested a full tablet report, sending...
I20240502 21:25:31.043288 26225 ts_manager.cc:194] Registered new tserver with Master: 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68:38255)
I20240502 21:25:31.044013 26225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50794
I20240502 21:25:31.053725 26566 raft_consensus.cc:482] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:31.053907 26566 raft_consensus.cc:504] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } }
I20240502 21:25:31.054502 26566 leader_election.cc:288] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67:35049), 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525)
I20240502 21:25:31.054911 26474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f57214b379402ca24433fc7a12df42" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" is_pre_election: true
I20240502 21:25:31.054931 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f57214b379402ca24433fc7a12df42" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" is_pre_election: true
I20240502 21:25:31.055167 26401 raft_consensus.cc:2379] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8570ae144d43dd95f9d6bf521ba178 in current term 1: Already voted for candidate ecce04fffd714216b8bebcff5ed2dc9e in this term.
I20240502 21:25:31.055533 26292 leader_election.cc:302] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8570ae144d43dd95f9d6bf521ba178; no voters: 4fc6cc31d05a40dcad5195a10cd0a485, ecce04fffd714216b8bebcff5ed2dc9e
I20240502 21:25:31.055797 26566 raft_consensus.cc:3037] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.057760 26566 raft_consensus.cc:2726] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:31.087568 26557 consensus_queue.cc:1035] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.091995 26531 consensus_queue.cc:1035] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.176673 26531 consensus_queue.cc:1035] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.180835 26558 consensus_queue.cc:1035] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.193555 26558 consensus_queue.cc:1035] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.196846 26557 consensus_queue.cc:1035] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.228984 26557 consensus_queue.cc:1035] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.232790 26558 consensus_queue.cc:1035] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.235142 26278 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:31.248458 26474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04f17dd6a0804a39a6748b673d55c0d0"
dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e"
mode: GRACEFUL
new_leader_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
from {username='slave'} at 127.0.0.1:57454
I20240502 21:25:31.248780 26474 raft_consensus.cc:595] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Received request to transfer leadership to TS 7a8570ae144d43dd95f9d6bf521ba178
I20240502 21:25:31.249917 26474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1277d64e9fc9404aa08c009dba377adf"
dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e"
mode: GRACEFUL
new_leader_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
from {username='slave'} at 127.0.0.1:57454
I20240502 21:25:31.250088 26474 raft_consensus.cc:595] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Received request to transfer leadership to TS 7a8570ae144d43dd95f9d6bf521ba178
I20240502 21:25:31.251063 26474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c80c249257f442598b2d3b7555fda8f5"
dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e"
mode: GRACEFUL
new_leader_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
from {username='slave'} at 127.0.0.1:57454
I20240502 21:25:31.251217 26474 raft_consensus.cc:595] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Received request to transfer leadership to TS 7a8570ae144d43dd95f9d6bf521ba178
I20240502 21:25:31.251571 26278 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 21:25:31.251883 26278 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:31.325753 26527 consensus_queue.cc:1035] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.328230 26558 consensus_queue.cc:1035] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.329991 26527 consensus_queue.cc:1035] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.331931 26531 raft_consensus.cc:979] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e: : Instructing follower 7a8570ae144d43dd95f9d6bf521ba178 to start an election
I20240502 21:25:31.332101 26531 raft_consensus.cc:1067] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Signalling peer 7a8570ae144d43dd95f9d6bf521ba178 to start an election
I20240502 21:25:31.334183 26325 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c80c249257f442598b2d3b7555fda8f5"
dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
from {username='slave'} at 127.0.0.1:39792
I20240502 21:25:31.334628 26325 raft_consensus.cc:482] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:31.334787 26325 raft_consensus.cc:3037] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:31.336155 26557 consensus_queue.cc:1035] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.336632 26563 consensus_queue.cc:1035] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.337694 26325 raft_consensus.cc:504] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:31.338784 26400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80c249257f442598b2d3b7555fda8f5" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485"
I20240502 21:25:31.340606 26474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80c249257f442598b2d3b7555fda8f5" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e"
I20240502 21:25:31.340799 26474 raft_consensus.cc:3032] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:31.341197 26325 leader_election.cc:288] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 2 election: Requested vote from peers 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525), ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67:35049)
I20240502 21:25:31.341238 26474 raft_consensus.cc:729] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Running, Role: LEADER
I20240502 21:25:31.341567 26474 consensus_queue.cc:260] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:31.341823 26527 consensus_queue.cc:1035] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:25:31.342047 26474 raft_consensus.cc:3037] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:31.345739 26474 raft_consensus.cc:2454] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8570ae144d43dd95f9d6bf521ba178 in term 2.
I20240502 21:25:31.346076 26293 leader_election.cc:302] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8570ae144d43dd95f9d6bf521ba178, ecce04fffd714216b8bebcff5ed2dc9e; no voters: 4fc6cc31d05a40dcad5195a10cd0a485
I20240502 21:25:31.346359 26566 raft_consensus.cc:2781] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:31.346732 26566 raft_consensus.cc:686] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 LEADER]: Becoming Leader. State: Replica: 7a8570ae144d43dd95f9d6bf521ba178, State: Running, Role: LEADER
I20240502 21:25:31.346982 26566 consensus_queue.cc:237] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:31.348757 26222 catalog_manager.cc:5484] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 reported cstate change: term changed from 1 to 2, leader changed from ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67) to 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65). New cstate: current_term: 2 leader_uuid: "7a8570ae144d43dd95f9d6bf521ba178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:31.372433 26559 consensus_queue.cc:1035] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.376380 26557 consensus_queue.cc:1035] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.379016 26559 raft_consensus.cc:979] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e: : Instructing follower 7a8570ae144d43dd95f9d6bf521ba178 to start an election
I20240502 21:25:31.379199 26557 raft_consensus.cc:1067] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Signalling peer 7a8570ae144d43dd95f9d6bf521ba178 to start an election
I20240502 21:25:31.379745 26325 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1277d64e9fc9404aa08c009dba377adf"
dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
from {username='slave'} at 127.0.0.1:39792
I20240502 21:25:31.379985 26325 raft_consensus.cc:482] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:31.380134 26325 raft_consensus.cc:3037] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:31.382175 26325 raft_consensus.cc:504] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:31.382701 26325 leader_election.cc:288] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 2 election: Requested vote from peers 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525), ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67:35049)
I20240502 21:25:31.383006 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1277d64e9fc9404aa08c009dba377adf" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485"
I20240502 21:25:31.383220 26401 raft_consensus.cc:3037] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:31.383175 26474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1277d64e9fc9404aa08c009dba377adf" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e"
I20240502 21:25:31.383360 26474 raft_consensus.cc:3032] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:31.383468 26474 raft_consensus.cc:729] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Running, Role: LEADER
I20240502 21:25:31.383735 26474 consensus_queue.cc:260] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:31.384037 26474 raft_consensus.cc:3037] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:31.385314 26401 raft_consensus.cc:2454] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8570ae144d43dd95f9d6bf521ba178 in term 2.
I20240502 21:25:31.385658 26292 leader_election.cc:302] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, 7a8570ae144d43dd95f9d6bf521ba178; no voters:
I20240502 21:25:31.385941 26566 raft_consensus.cc:2781] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:31.385969 26474 raft_consensus.cc:2454] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8570ae144d43dd95f9d6bf521ba178 in term 2.
I20240502 21:25:31.386085 26566 raft_consensus.cc:686] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 LEADER]: Becoming Leader. State: Replica: 7a8570ae144d43dd95f9d6bf521ba178, State: Running, Role: LEADER
I20240502 21:25:31.386314 26566 consensus_queue.cc:237] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:31.388188 26225 catalog_manager.cc:5484] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 reported cstate change: term changed from 1 to 2, leader changed from ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67) to 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65). New cstate: current_term: 2 leader_uuid: "7a8570ae144d43dd95f9d6bf521ba178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:31.661998 26557 raft_consensus.cc:979] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e: : Instructing follower 7a8570ae144d43dd95f9d6bf521ba178 to start an election
I20240502 21:25:31.662175 26557 raft_consensus.cc:1067] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Signalling peer 7a8570ae144d43dd95f9d6bf521ba178 to start an election
I20240502 21:25:31.662703 26325 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04f17dd6a0804a39a6748b673d55c0d0"
dest_uuid: "7a8570ae144d43dd95f9d6bf521ba178"
from {username='slave'} at 127.0.0.1:39792
I20240502 21:25:31.662875 26325 raft_consensus.cc:482] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:31.663009 26325 raft_consensus.cc:3037] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:31.664893 26325 raft_consensus.cc:504] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:31.665520 26325 leader_election.cc:288] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 2 election: Requested vote from peers 4fc6cc31d05a40dcad5195a10cd0a485 (127.25.16.66:37525), ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67:35049)
I20240502 21:25:31.665817 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f17dd6a0804a39a6748b673d55c0d0" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4fc6cc31d05a40dcad5195a10cd0a485"
I20240502 21:25:31.666006 26401 raft_consensus.cc:3037] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:31.665946 26474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f17dd6a0804a39a6748b673d55c0d0" candidate_uuid: "7a8570ae144d43dd95f9d6bf521ba178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ecce04fffd714216b8bebcff5ed2dc9e"
I20240502 21:25:31.666144 26474 raft_consensus.cc:3032] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:31.666240 26474 raft_consensus.cc:729] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecce04fffd714216b8bebcff5ed2dc9e, State: Running, Role: LEADER
I20240502 21:25:31.666492 26474 consensus_queue.cc:260] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:31.666795 26474 raft_consensus.cc:3037] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:31.668066 26401 raft_consensus.cc:2454] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8570ae144d43dd95f9d6bf521ba178 in term 2.
I20240502 21:25:31.668489 26292 leader_election.cc:302] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc6cc31d05a40dcad5195a10cd0a485, 7a8570ae144d43dd95f9d6bf521ba178; no voters:
I20240502 21:25:31.668766 26667 raft_consensus.cc:2781] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:31.668782 26474 raft_consensus.cc:2454] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8570ae144d43dd95f9d6bf521ba178 in term 2.
I20240502 21:25:31.668900 26667 raft_consensus.cc:686] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 LEADER]: Becoming Leader. State: Replica: 7a8570ae144d43dd95f9d6bf521ba178, State: Running, Role: LEADER
I20240502 21:25:31.669165 26667 consensus_queue.cc:237] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } }
I20240502 21:25:31.670996 26225 catalog_manager.cc:5484] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 reported cstate change: term changed from 1 to 2, leader changed from ecce04fffd714216b8bebcff5ed2dc9e (127.25.16.67) to 7a8570ae144d43dd95f9d6bf521ba178 (127.25.16.65). New cstate: current_term: 2 leader_uuid: "7a8570ae144d43dd95f9d6bf521ba178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:31.784255 26474 raft_consensus.cc:1261] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Refusing update from remote peer 7a8570ae144d43dd95f9d6bf521ba178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:25:31.784833 26667 consensus_queue.cc:1035] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:31.787591 26401 raft_consensus.cc:1261] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Refusing update from remote peer 7a8570ae144d43dd95f9d6bf521ba178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:25:31.788108 26667 consensus_queue.cc:1035] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:31.908792 26475 raft_consensus.cc:1261] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Refusing update from remote peer 7a8570ae144d43dd95f9d6bf521ba178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:25:31.909315 26667 consensus_queue.cc:1035] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:31.912081 26401 raft_consensus.cc:3037] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:31.914911 26401 raft_consensus.cc:1261] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Refusing update from remote peer 7a8570ae144d43dd95f9d6bf521ba178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:25:31.915467 26566 consensus_queue.cc:1035] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:32.045142 26644 heartbeater.cc:503] Master 127.25.16.126:32883 was elected leader, sending a full tablet report...
I20240502 21:25:32.061724 26475 raft_consensus.cc:1261] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Refusing update from remote peer 7a8570ae144d43dd95f9d6bf521ba178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:25:32.062227 26566 consensus_queue.cc:1035] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:32.065093 26401 raft_consensus.cc:1261] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Refusing update from remote peer 7a8570ae144d43dd95f9d6bf521ba178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:25:32.065722 26670 consensus_queue.cc:1035] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:33.252246 26278 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:33.253633 26278 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:25:33.253775 26278 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:35.254174 26278 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:35.255671 26278 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:25:35.255839 26278 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:36.246031 26325 consensus_queue.cc:237] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } attrs { replace: true } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } }
I20240502 21:25:36.248462 26401 raft_consensus.cc:1261] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Refusing update from remote peer 7a8570ae144d43dd95f9d6bf521ba178: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:25:36.249145 26676 consensus_queue.cc:1035] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:25:36.249863 26475 raft_consensus.cc:1261] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Refusing update from remote peer 7a8570ae144d43dd95f9d6bf521ba178: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:25:36.250342 26676 consensus_queue.cc:1035] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:25:36.252214 26676 raft_consensus.cc:2932] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } attrs { replace: true } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.252621 26401 raft_consensus.cc:2932] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } attrs { replace: true } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.254446 26475 raft_consensus.cc:2932] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } attrs { replace: true } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.258405 26225 catalog_manager.cc:5484] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 reported cstate change: config changed from index -1 to 3, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New cstate: current_term: 2 leader_uuid: "7a8570ae144d43dd95f9d6bf521ba178" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:36.262130 26290 consensus_peers.cc:485] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 -> Peer 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68:38255): Couldn't send request to peer 2cb9a05822ba4213a35821b97abdaf06. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c80c249257f442598b2d3b7555fda8f5. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:36.267570 26401 consensus_queue.cc:237] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } }
I20240502 21:25:36.270503 26325 raft_consensus.cc:1261] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Refusing update from remote peer 4fc6cc31d05a40dcad5195a10cd0a485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:36.270535 26475 raft_consensus.cc:1261] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Refusing update from remote peer 4fc6cc31d05a40dcad5195a10cd0a485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:36.271046 26527 consensus_queue.cc:1035] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:36.271382 26687 consensus_queue.cc:1035] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:36.274051 26527 raft_consensus.cc:2932] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
W20240502 21:25:36.276731 26365 consensus_peers.cc:485] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 -> Peer 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68:38255): Couldn't send request to peer 2cb9a05822ba4213a35821b97abdaf06. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8328927a71f0468fb874133b3df2ae42. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:36.276544 26475 raft_consensus.cc:2932] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.281678 26222 catalog_manager.cc:5484] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e reported cstate change: config changed from index -1 to 2, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New cstate: current_term: 1 leader_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.294457 26325 raft_consensus.cc:2932] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.297369 26475 consensus_queue.cc:237] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } attrs { replace: true } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } }
I20240502 21:25:36.301286 26325 raft_consensus.cc:1261] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Refusing update from remote peer ecce04fffd714216b8bebcff5ed2dc9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:36.301398 26401 raft_consensus.cc:1261] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Refusing update from remote peer ecce04fffd714216b8bebcff5ed2dc9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:36.301971 26694 consensus_queue.cc:1035] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:36.302553 26694 consensus_queue.cc:1035] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:36.305315 26694 raft_consensus.cc:2932] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } attrs { replace: true } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.305752 26401 raft_consensus.cc:2932] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } attrs { replace: true } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.306088 26325 raft_consensus.cc:2932] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } attrs { replace: true } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
W20240502 21:25:36.310994 26439 consensus_peers.cc:485] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e -> Peer 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68:38255): Couldn't send request to peer 2cb9a05822ba4213a35821b97abdaf06. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7723674bc40a4df5a9dea66778d72236. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:36.311352 26222 catalog_manager.cc:5484] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178 reported cstate change: config changed from index -1 to 2, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New cstate: current_term: 1 leader_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } attrs { replace: true } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.312417 26401 consensus_queue.cc:237] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } }
I20240502 21:25:36.314819 26325 raft_consensus.cc:1261] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Refusing update from remote peer 4fc6cc31d05a40dcad5195a10cd0a485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:36.314896 26475 raft_consensus.cc:1261] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Refusing update from remote peer 4fc6cc31d05a40dcad5195a10cd0a485: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:36.315433 26527 consensus_queue.cc:1035] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:25:36.315722 26365 consensus_peers.cc:485] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 -> Peer 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68:38255): Couldn't send request to peer 2cb9a05822ba4213a35821b97abdaf06. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 783f9c409f6543dfa54f7d764fd9b2db. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:36.316112 26691 consensus_queue.cc:1035] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:36.318506 26527 raft_consensus.cc:2932] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.319089 26325 raft_consensus.cc:2932] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.319387 26475 raft_consensus.cc:2932] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.323451 26225 catalog_manager.cc:5484] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e reported cstate change: config changed from index -1 to 2, NON_VOTER 2cb9a05822ba4213a35821b97abdaf06 (127.25.16.68) added. New cstate: current_term: 1 leader_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7a8570ae144d43dd95f9d6bf521ba178" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33987 } } peers { permanent_uuid: "4fc6cc31d05a40dcad5195a10cd0a485" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37525 } } peers { permanent_uuid: "ecce04fffd714216b8bebcff5ed2dc9e" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35049 } attrs { replace: true } } peers { permanent_uuid: "2cb9a05822ba4213a35821b97abdaf06" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 38255 } attrs { promote: true } } }
I20240502 21:25:36.550071 25665 tablet_server.cc:178] TabletServer@127.25.16.65:0 shutting down...
W20240502 21:25:36.555372 26211 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20240502 21:25:36.557533 26273 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 21:25:36.557973 26273 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 21:25:36.559715 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:25:36.560051 25665 tablet_replica.cc:330] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178: stopping tablet replica
I20240502 21:25:36.560272 25665 raft_consensus.cc:2229] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.560501 25665 raft_consensus.cc:2258] T e9f57214b379402ca24433fc7a12df42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.561169 25665 tablet_replica.cc:330] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178: stopping tablet replica
I20240502 21:25:36.561378 25665 raft_consensus.cc:2229] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 LEADER]: Raft consensus shutting down.
I20240502 21:25:36.561726 25665 raft_consensus.cc:2258] T 04f17dd6a0804a39a6748b673d55c0d0 P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.562285 25665 tablet_replica.cc:330] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178: stopping tablet replica
I20240502 21:25:36.562451 25665 raft_consensus.cc:2229] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.562696 25665 raft_consensus.cc:2258] T 783f9c409f6543dfa54f7d764fd9b2db P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.563452 25665 tablet_replica.cc:330] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178: stopping tablet replica
I20240502 21:25:36.563700 25665 raft_consensus.cc:2229] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 LEADER]: Raft consensus shutting down.
I20240502 21:25:36.564002 25665 raft_consensus.cc:2258] T 1277d64e9fc9404aa08c009dba377adf P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.564528 25665 tablet_replica.cc:330] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178: stopping tablet replica
I20240502 21:25:36.564690 25665 raft_consensus.cc:2229] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.564857 25665 raft_consensus.cc:2258] T 8042e4ef446943f1be34424b94260767 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.565351 25665 tablet_replica.cc:330] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178: stopping tablet replica
I20240502 21:25:36.565526 25665 raft_consensus.cc:2229] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.565721 25665 raft_consensus.cc:2258] T 8328927a71f0468fb874133b3df2ae42 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.566313 25665 tablet_replica.cc:330] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178: stopping tablet replica
I20240502 21:25:36.566457 25665 raft_consensus.cc:2229] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 LEADER]: Raft consensus shutting down.
I20240502 21:25:36.566781 25665 raft_consensus.cc:2258] T c80c249257f442598b2d3b7555fda8f5 P 7a8570ae144d43dd95f9d6bf521ba178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.567420 25665 tablet_replica.cc:330] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178: stopping tablet replica
I20240502 21:25:36.567574 25665 raft_consensus.cc:2229] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.567747 25665 raft_consensus.cc:2258] T 7723674bc40a4df5a9dea66778d72236 P 7a8570ae144d43dd95f9d6bf521ba178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.584931 25665 tablet_server.cc:194] TabletServer@127.25.16.65:0 shutdown complete.
I20240502 21:25:36.589267 25665 tablet_server.cc:178] TabletServer@127.25.16.66:0 shutting down...
W20240502 21:25:36.591850 26273 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 21:25:36.592316 26273 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 21:25:36.594748 26273 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 21:25:36.594841 26273 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 21:25:36.598352 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:25:36.598587 25665 tablet_replica.cc:330] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485: stopping tablet replica
I20240502 21:25:36.598872 25665 raft_consensus.cc:2229] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.599094 25665 raft_consensus.cc:2258] T 04f17dd6a0804a39a6748b673d55c0d0 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.599686 25665 tablet_replica.cc:330] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485: stopping tablet replica
I20240502 21:25:36.599850 25665 raft_consensus.cc:2229] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:36.600188 25665 raft_consensus.cc:2258] T 8328927a71f0468fb874133b3df2ae42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.600921 25665 tablet_replica.cc:330] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485: stopping tablet replica
I20240502 21:25:36.601087 25665 raft_consensus.cc:2229] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.601280 25665 raft_consensus.cc:2258] T c80c249257f442598b2d3b7555fda8f5 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.601989 25665 tablet_replica.cc:330] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485: stopping tablet replica
I20240502 21:25:36.602145 25665 raft_consensus.cc:2229] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.602334 25665 raft_consensus.cc:2258] T e9f57214b379402ca24433fc7a12df42 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.603050 25665 tablet_replica.cc:330] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485: stopping tablet replica
I20240502 21:25:36.603238 25665 raft_consensus.cc:2229] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.603447 25665 raft_consensus.cc:2258] T 7723674bc40a4df5a9dea66778d72236 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.604161 25665 tablet_replica.cc:330] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485: stopping tablet replica
I20240502 21:25:36.604368 25665 raft_consensus.cc:2229] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:36.604754 25665 raft_consensus.cc:2258] T 783f9c409f6543dfa54f7d764fd9b2db P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.605612 25665 tablet_replica.cc:330] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485: stopping tablet replica
I20240502 21:25:36.605809 25665 raft_consensus.cc:2229] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.605988 25665 raft_consensus.cc:2258] T 1277d64e9fc9404aa08c009dba377adf P 4fc6cc31d05a40dcad5195a10cd0a485 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.606523 25665 tablet_replica.cc:330] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485: stopping tablet replica
I20240502 21:25:36.606685 25665 raft_consensus.cc:2229] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.606873 25665 raft_consensus.cc:2258] T 8042e4ef446943f1be34424b94260767 P 4fc6cc31d05a40dcad5195a10cd0a485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.624198 25665 tablet_server.cc:194] TabletServer@127.25.16.66:0 shutdown complete.
I20240502 21:25:36.628806 25665 tablet_server.cc:178] TabletServer@127.25.16.67:0 shutting down...
W20240502 21:25:36.631732 26273 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 21:25:36.631996 26273 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 21:25:36.636639 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:25:36.636857 25665 tablet_replica.cc:330] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e: stopping tablet replica
I20240502 21:25:36.637056 25665 raft_consensus.cc:2229] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.637272 25665 raft_consensus.cc:2258] T 783f9c409f6543dfa54f7d764fd9b2db P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.637928 25665 tablet_replica.cc:330] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e: stopping tablet replica
I20240502 21:25:36.638088 25665 raft_consensus.cc:2229] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:36.638401 25665 raft_consensus.cc:2258] T 7723674bc40a4df5a9dea66778d72236 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.638983 25665 tablet_replica.cc:330] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e: stopping tablet replica
I20240502 21:25:36.639120 25665 raft_consensus.cc:2229] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:36.639366 25665 raft_consensus.cc:2258] T e9f57214b379402ca24433fc7a12df42 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.639760 25665 tablet_replica.cc:330] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e: stopping tablet replica
I20240502 21:25:36.639917 25665 raft_consensus.cc:2229] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.640079 25665 raft_consensus.cc:2258] T c80c249257f442598b2d3b7555fda8f5 P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.640702 25665 tablet_replica.cc:330] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e: stopping tablet replica
I20240502 21:25:36.640830 25665 raft_consensus.cc:2229] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.641002 25665 raft_consensus.cc:2258] T 8328927a71f0468fb874133b3df2ae42 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.641573 25665 tablet_replica.cc:330] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e: stopping tablet replica
I20240502 21:25:36.641717 25665 raft_consensus.cc:2229] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.641857 25665 raft_consensus.cc:2258] T 1277d64e9fc9404aa08c009dba377adf P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.642257 25665 tablet_replica.cc:330] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e: stopping tablet replica
I20240502 21:25:36.642381 25665 raft_consensus.cc:2229] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:36.642617 25665 raft_consensus.cc:2258] T 8042e4ef446943f1be34424b94260767 P ecce04fffd714216b8bebcff5ed2dc9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.643010 25665 tablet_replica.cc:330] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e: stopping tablet replica
I20240502 21:25:36.643146 25665 raft_consensus.cc:2229] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:36.643275 25665 raft_consensus.cc:2258] T 04f17dd6a0804a39a6748b673d55c0d0 P ecce04fffd714216b8bebcff5ed2dc9e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.658106 25665 tablet_server.cc:194] TabletServer@127.25.16.67:0 shutdown complete.
I20240502 21:25:36.661917 25665 tablet_server.cc:178] TabletServer@127.25.16.68:0 shutting down...
I20240502 21:25:36.669778 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:25:36.682346 25665 tablet_server.cc:194] TabletServer@127.25.16.68:0 shutdown complete.
I20240502 21:25:36.684935 25665 master.cc:537] Master@127.25.16.126:32883 shutting down...
I20240502 21:25:36.691395 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:36.691606 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:36.691700 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8bdd8e5ce50549cc8553aba7652a17c2: stopping tablet replica
I20240502 21:25:36.705102 25665 master.cc:552] Master@127.25.16.126:32883 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7542 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240502 21:25:36.720317 25665 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.16.126:41201,127.25.16.125:42649,127.25.16.124:42843
I20240502 21:25:36.720876 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:36.723225 26703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.723511 26704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.723799 26706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.724007 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:36.724272 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:36.724395 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:36.724467 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685136724431 us; error 0 us; skew 500 ppm
I20240502 21:25:36.724622 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:36.725790 25665 webserver.cc:460] Webserver started at http://127.25.16.126:44129/ using document root <none> and password file <none>
I20240502 21:25:36.725998 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:36.726084 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:36.726217 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:36.726858 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-0-root/instance:
uuid: "0102d924eb4b4051b3f4fafb85945f0f"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-b524"
I20240502 21:25:36.729319 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:36.730844 26711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.731209 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 21:25:36.731304 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-0-root
uuid: "0102d924eb4b4051b3f4fafb85945f0f"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-b524"
I20240502 21:25:36.731417 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:36.741048 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:36.741531 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:36.753171 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.126:41201
I20240502 21:25:36.753193 26762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.126:41201 every 8 connection(s)
I20240502 21:25:36.754338 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 21:25:36.754428 26763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:25:36.756856 26766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.757174 26765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.758041 26763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } has no permanent_uuid. Determining permanent_uuid...
I20240502 21:25:36.758188 25665 server_base.cc:1034] running on GCE node
W20240502 21:25:36.758339 26768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.758806 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:36.758883 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:36.758987 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685136758947 us; error 0 us; skew 500 ppm
I20240502 21:25:36.759150 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:36.761016 25665 webserver.cc:460] Webserver started at http://127.25.16.125:38475/ using document root <none> and password file <none>
I20240502 21:25:36.761216 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:36.761309 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:36.761430 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:36.761996 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-1-root/instance:
uuid: "66ea7a8ca11649e1a69ea248a450c8b1"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-b524"
I20240502 21:25:36.764488 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:36.766613 26775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.767011 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 21:25:36.767113 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-1-root
uuid: "66ea7a8ca11649e1a69ea248a450c8b1"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-b524"
I20240502 21:25:36.767241 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:36.771245 26763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } has no permanent_uuid. Determining permanent_uuid...
W20240502 21:25:36.773100 26763 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.25.16.125:42649: Network error: Client connection negotiation failed: client connection to 127.25.16.125:42649: connect: Connection refused (error 111)
I20240502 21:25:36.792670 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:36.793200 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:36.805953 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.125:42649
I20240502 21:25:36.806006 26827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.125:42649 every 8 connection(s)
I20240502 21:25:36.807648 26828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:36.807713 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:36.810501 26830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.810576 26831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.811338 26833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.811307 26828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } has no permanent_uuid. Determining permanent_uuid...
I20240502 21:25:36.812274 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:36.812557 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:36.812646 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:36.812722 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685136812685 us; error 0 us; skew 500 ppm
I20240502 21:25:36.812899 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:36.814342 25665 webserver.cc:460] Webserver started at http://127.25.16.124:43795/ using document root <none> and password file <none>
I20240502 21:25:36.814584 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:36.814694 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:36.814872 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:36.815486 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-2-root/instance:
uuid: "22d8c50ce69045d099d53c704c434a84"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-b524"
I20240502 21:25:36.817555 26828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } has no permanent_uuid. Determining permanent_uuid...
I20240502 21:25:36.818794 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:25:36.819293 26763 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } attempt: 1
I20240502 21:25:36.820962 26841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.821442 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 21:25:36.821568 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-2-root
uuid: "22d8c50ce69045d099d53c704c434a84"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-b524"
I20240502 21:25:36.821766 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:36.825385 26828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } has no permanent_uuid. Determining permanent_uuid...
I20240502 21:25:36.826308 26763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } has no permanent_uuid. Determining permanent_uuid...
W20240502 21:25:36.827261 26828 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.25.16.124:42843: Network error: Client connection negotiation failed: client connection to 127.25.16.124:42843: connect: Connection refused (error 111)
W20240502 21:25:36.827729 26763 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.25.16.124:42843: Network error: Client connection negotiation failed: client connection to 127.25.16.124:42843: connect: Connection refused (error 111)
I20240502 21:25:36.836866 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:36.837349 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:36.848484 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.124:42843
I20240502 21:25:36.848533 26893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.124:42843 every 8 connection(s)
I20240502 21:25:36.849372 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:25:36.849757 26894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:36.851960 26894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } has no permanent_uuid. Determining permanent_uuid...
I20240502 21:25:36.855463 26828 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } attempt: 1
I20240502 21:25:36.857842 26894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } has no permanent_uuid. Determining permanent_uuid...
I20240502 21:25:36.859932 26763 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } attempt: 1
I20240502 21:25:36.863818 26894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } has no permanent_uuid. Determining permanent_uuid...
I20240502 21:25:36.865146 26828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1: Bootstrap starting.
I20240502 21:25:36.866839 26828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.869004 26828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1: No bootstrap required, opened a new log
I20240502 21:25:36.869877 26763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f: Bootstrap starting.
I20240502 21:25:36.869869 26828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } }
I20240502 21:25:36.870106 26828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.870198 26828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ea7a8ca11649e1a69ea248a450c8b1, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.870496 26828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } }
I20240502 21:25:36.871564 26763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.872030 26894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84: Bootstrap starting.
I20240502 21:25:36.872612 26900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } } }
I20240502 21:25:36.872993 26828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:36.872998 26900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:36.873519 26894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.875106 26894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84: No bootstrap required, opened a new log
I20240502 21:25:36.875711 26894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } }
I20240502 21:25:36.875880 26894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.875965 26894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22d8c50ce69045d099d53c704c434a84, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.876197 26894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } }
I20240502 21:25:36.876883 26904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } } }
I20240502 21:25:36.877174 26894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:36.877207 26904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:36.877348 26763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f: No bootstrap required, opened a new log
I20240502 21:25:36.878121 26763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } }
I20240502 21:25:36.878350 26763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.878446 26763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0102d924eb4b4051b3f4fafb85945f0f, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.878706 26763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } }
I20240502 21:25:36.879475 26907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } } }
I20240502 21:25:36.879792 26763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:36.879786 26907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:36.883431 26904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:36.883594 26904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } }
I20240502 21:25:36.884557 26904 leader_election.cc:288] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0102d924eb4b4051b3f4fafb85945f0f (127.25.16.126:41201), 66ea7a8ca11649e1a69ea248a450c8b1 (127.25.16.125:42649)
I20240502 21:25:36.885143 26738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22d8c50ce69045d099d53c704c434a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102d924eb4b4051b3f4fafb85945f0f" is_pre_election: true
I20240502 21:25:36.885300 26803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22d8c50ce69045d099d53c704c434a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" is_pre_election: true
I20240502 21:25:36.885462 26738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22d8c50ce69045d099d53c704c434a84 in term 0.
I20240502 21:25:36.885504 26803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22d8c50ce69045d099d53c704c434a84 in term 0.
I20240502 21:25:36.885937 26843 leader_election.cc:302] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102d924eb4b4051b3f4fafb85945f0f, 22d8c50ce69045d099d53c704c434a84; no voters:
I20240502 21:25:36.886206 26904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:36.886368 26904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:36.886487 26904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:36.888445 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240502 21:25:36.888897 26904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } }
W20240502 21:25:36.889806 26927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240502 21:25:36.889783 26738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22d8c50ce69045d099d53c704c434a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0102d924eb4b4051b3f4fafb85945f0f"
W20240502 21:25:36.889941 26927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 21:25:36.889992 26738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:36.890668 26904 leader_election.cc:288] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [CANDIDATE]: Term 1 election: Requested vote from peers 0102d924eb4b4051b3f4fafb85945f0f (127.25.16.126:41201), 66ea7a8ca11649e1a69ea248a450c8b1 (127.25.16.125:42649)
I20240502 21:25:36.891525 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 21:25:36.892143 26803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22d8c50ce69045d099d53c704c434a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ea7a8ca11649e1a69ea248a450c8b1"
I20240502 21:25:36.892359 26803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:36.893497 26738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22d8c50ce69045d099d53c704c434a84 in term 1.
I20240502 21:25:36.893975 26843 leader_election.cc:302] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102d924eb4b4051b3f4fafb85945f0f, 22d8c50ce69045d099d53c704c434a84; no voters:
I20240502 21:25:36.894321 26904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:36.894745 26904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 1 LEADER]: Becoming Leader. State: Replica: 22d8c50ce69045d099d53c704c434a84, State: Running, Role: LEADER
I20240502 21:25:36.895138 26803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22d8c50ce69045d099d53c704c434a84 in term 1.
I20240502 21:25:36.895043 26904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } }
I20240502 21:25:36.895267 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:36.895988 26929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 21:25:36.896093 26929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 21:25:36.896608 26904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22d8c50ce69045d099d53c704c434a84. Latest consensus state: current_term: 1 leader_uuid: "22d8c50ce69045d099d53c704c434a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } } }
I20240502 21:25:36.896840 26904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [sys.catalog]: This master's current role is: LEADER
W20240502 21:25:36.899144 26937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 21:25:36.899259 26937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 21:25:36.899452 26940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.900333 26942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.903260 26944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.905400 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:36.905652 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:36.905732 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:36.905805 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685136905771 us; error 0 us; skew 500 ppm
I20240502 21:25:36.906013 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:36.906118 26941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:36.907435 25665 webserver.cc:460] Webserver started at http://127.25.16.65:46241/ using document root <none> and password file <none>
I20240502 21:25:36.907724 26941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:36.907734 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:36.907853 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:36.908025 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:36.908833 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-0-root/instance:
uuid: "d084f683b4274a438a4b1f8a4deddc74"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-b524"
I20240502 21:25:36.910766 26803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [term 1 FOLLOWER]: Refusing update from remote peer 22d8c50ce69045d099d53c704c434a84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:36.910768 26738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 1 FOLLOWER]: Refusing update from remote peer 22d8c50ce69045d099d53c704c434a84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:36.911335 26938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:36.911911 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:25:36.911836 26904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:36.914160 26907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22d8c50ce69045d099d53c704c434a84. Latest consensus state: current_term: 1 leader_uuid: "22d8c50ce69045d099d53c704c434a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } } }
I20240502 21:25:36.914422 26907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:36.915632 26900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22d8c50ce69045d099d53c704c434a84. Latest consensus state: current_term: 1 leader_uuid: "22d8c50ce69045d099d53c704c434a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } } }
I20240502 21:25:36.915899 26900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:36.918139 26900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22d8c50ce69045d099d53c704c434a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } } }
I20240502 21:25:36.918344 26900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:36.918535 26941 catalog_manager.cc:1347] Generated new cluster ID: 36ca7bf707fb4e7b9acac4bda6bbf37f
I20240502 21:25:36.918613 26941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:36.920783 26907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22d8c50ce69045d099d53c704c434a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } } }
I20240502 21:25:36.921001 26907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [sys.catalog]: This master's current role is: FOLLOWER
I20240502 21:25:36.921358 26953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.921773 25665 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.002s
I20240502 21:25:36.921897 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-0-root
uuid: "d084f683b4274a438a4b1f8a4deddc74"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-b524"
I20240502 21:25:36.922034 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:36.924741 26938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22d8c50ce69045d099d53c704c434a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } } }
I20240502 21:25:36.924989 26938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:36.925268 26904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22d8c50ce69045d099d53c704c434a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } } }
I20240502 21:25:36.925426 26904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:36.932814 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:36.933648 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:36.934389 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:36.935686 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:36.935802 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.935902 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:36.935985 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.936959 26941 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:36.937819 26941 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:36.949435 26941 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84: Generated new TSK 0
I20240502 21:25:36.949671 26941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:36.956084 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.65:42389
I20240502 21:25:36.956135 27020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.65:42389 every 8 connection(s)
I20240502 21:25:36.965858 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 21:25:36.967972 27021 heartbeater.cc:347] Connected to a master server at 127.25.16.124:42843
I20240502 21:25:36.968156 27021 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:36.968891 27021 heartbeater.cc:511] Master 127.25.16.124:42843 requested a full tablet report, sending...
I20240502 21:25:36.969918 26859 ts_manager.cc:194] Registered new tserver with Master: d084f683b4274a438a4b1f8a4deddc74 (127.25.16.65:42389)
I20240502 21:25:36.971500 26859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39612
W20240502 21:25:36.973040 27028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.973697 27023 heartbeater.cc:347] Connected to a master server at 127.25.16.125:42649
I20240502 21:25:36.973836 27023 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:36.974051 27022 heartbeater.cc:347] Connected to a master server at 127.25.16.126:41201
I20240502 21:25:36.974155 27023 heartbeater.cc:511] Master 127.25.16.125:42649 requested a full tablet report, sending...
I20240502 21:25:36.974237 27022 heartbeater.cc:464] Registering TS with master...
W20240502 21:25:36.974263 27029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.974507 27022 heartbeater.cc:511] Master 127.25.16.126:41201 requested a full tablet report, sending...
I20240502 21:25:36.975327 26793 ts_manager.cc:194] Registered new tserver with Master: d084f683b4274a438a4b1f8a4deddc74 (127.25.16.65:42389)
I20240502 21:25:36.975493 26728 ts_manager.cc:194] Registered new tserver with Master: d084f683b4274a438a4b1f8a4deddc74 (127.25.16.65:42389)
W20240502 21:25:36.977864 27031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.978070 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:36.978338 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:36.978416 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:36.978487 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685136978451 us; error 0 us; skew 500 ppm
I20240502 21:25:36.978631 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:36.979773 25665 webserver.cc:460] Webserver started at http://127.25.16.66:44219/ using document root <none> and password file <none>
I20240502 21:25:36.979977 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:36.980064 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:36.980185 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:36.980808 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-1-root/instance:
uuid: "2d405a32679f4da486c0fd6362ff108d"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-b524"
I20240502 21:25:36.983150 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:36.984673 27036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.984992 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:25:36.985107 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-1-root
uuid: "2d405a32679f4da486c0fd6362ff108d"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-b524"
I20240502 21:25:36.985226 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:36.996747 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:36.997220 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:36.997665 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:36.998452 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:36.998514 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.998589 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:36.998631 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.012537 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.66:33535
I20240502 21:25:37.012580 27098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.66:33535 every 8 connection(s)
I20240502 21:25:37.015478 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 21:25:37.024132 27099 heartbeater.cc:347] Connected to a master server at 127.25.16.124:42843
I20240502 21:25:37.024348 27099 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:37.024915 27099 heartbeater.cc:511] Master 127.25.16.124:42843 requested a full tablet report, sending...
I20240502 21:25:37.025970 26859 ts_manager.cc:194] Registered new tserver with Master: 2d405a32679f4da486c0fd6362ff108d (127.25.16.66:33535)
I20240502 21:25:37.027139 26859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39614
I20240502 21:25:37.028434 27100 heartbeater.cc:347] Connected to a master server at 127.25.16.126:41201
I20240502 21:25:37.028564 27100 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:37.028904 27100 heartbeater.cc:511] Master 127.25.16.126:41201 requested a full tablet report, sending...
I20240502 21:25:37.029996 27101 heartbeater.cc:347] Connected to a master server at 127.25.16.125:42649
I20240502 21:25:37.030182 27101 heartbeater.cc:464] Registering TS with master...
W20240502 21:25:37.030261 27106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:37.030553 27101 heartbeater.cc:511] Master 127.25.16.125:42649 requested a full tablet report, sending...
I20240502 21:25:37.030553 26728 ts_manager.cc:194] Registered new tserver with Master: 2d405a32679f4da486c0fd6362ff108d (127.25.16.66:33535)
W20240502 21:25:37.030957 27107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:37.031817 26793 ts_manager.cc:194] Registered new tserver with Master: 2d405a32679f4da486c0fd6362ff108d (127.25.16.66:33535)
W20240502 21:25:37.032555 27109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:37.033120 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:37.033354 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:37.033432 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:37.033489 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685137033472 us; error 0 us; skew 500 ppm
I20240502 21:25:37.033641 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:37.034771 25665 webserver.cc:460] Webserver started at http://127.25.16.67:35077/ using document root <none> and password file <none>
I20240502 21:25:37.035005 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:37.035080 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:37.035205 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:37.035791 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-2-root/instance:
uuid: "15a040f981d7429cbfd7669406e803a7"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-b524"
I20240502 21:25:37.038177 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 21:25:37.039696 27114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.040055 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 21:25:37.040150 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-2-root
uuid: "15a040f981d7429cbfd7669406e803a7"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-b524"
I20240502 21:25:37.040243 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714685126025081-25665-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:37.056037 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:37.056635 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:37.057181 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:37.058017 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:37.058096 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.058168 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:37.058221 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.073161 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.67:40933
I20240502 21:25:37.073199 27176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.67:40933 every 8 connection(s)
I20240502 21:25:37.082897 27178 heartbeater.cc:347] Connected to a master server at 127.25.16.126:41201
I20240502 21:25:37.083084 27178 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:37.083205 27177 heartbeater.cc:347] Connected to a master server at 127.25.16.124:42843
I20240502 21:25:37.083345 27177 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:37.083703 27178 heartbeater.cc:511] Master 127.25.16.126:41201 requested a full tablet report, sending...
I20240502 21:25:37.083865 27177 heartbeater.cc:511] Master 127.25.16.124:42843 requested a full tablet report, sending...
I20240502 21:25:37.084705 26859 ts_manager.cc:194] Registered new tserver with Master: 15a040f981d7429cbfd7669406e803a7 (127.25.16.67:40933)
I20240502 21:25:37.084885 26728 ts_manager.cc:194] Registered new tserver with Master: 15a040f981d7429cbfd7669406e803a7 (127.25.16.67:40933)
I20240502 21:25:37.085588 26859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39616
I20240502 21:25:37.085888 27179 heartbeater.cc:347] Connected to a master server at 127.25.16.125:42649
I20240502 21:25:37.086040 27179 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:37.086493 27179 heartbeater.cc:511] Master 127.25.16.125:42649 requested a full tablet report, sending...
I20240502 21:25:37.087311 26793 ts_manager.cc:194] Registered new tserver with Master: 15a040f981d7429cbfd7669406e803a7 (127.25.16.67:40933)
I20240502 21:25:37.087963 25665 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012048698s
I20240502 21:25:37.891146 26927 catalog_manager.cc:1259] Loaded cluster ID: 36ca7bf707fb4e7b9acac4bda6bbf37f
I20240502 21:25:37.891247 26927 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f: loading cluster ID for follower catalog manager: success
I20240502 21:25:37.893150 26927 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f: acquiring CA information for follower catalog manager: success
I20240502 21:25:37.894111 26927 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 21:25:37.897099 26929 catalog_manager.cc:1259] Loaded cluster ID: 36ca7bf707fb4e7b9acac4bda6bbf37f
I20240502 21:25:37.897166 26929 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1: loading cluster ID for follower catalog manager: success
I20240502 21:25:37.898818 26929 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1: acquiring CA information for follower catalog manager: success
I20240502 21:25:37.899699 26929 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 21:25:37.973040 27021 heartbeater.cc:503] Master 127.25.16.124:42843 was elected leader, sending a full tablet report...
I20240502 21:25:38.028750 27099 heartbeater.cc:503] Master 127.25.16.124:42843 was elected leader, sending a full tablet report...
I20240502 21:25:38.086894 27177 heartbeater.cc:503] Master 127.25.16.124:42843 was elected leader, sending a full tablet report...
I20240502 21:25:38.894534 26936 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:39.146624 25665 test_util.cc:268] Using random seed: 1050597226
I20240502 21:25:39.179710 26859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39624:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 21:25:39.202423 26986 tablet_service.cc:1467] Processing CreateTablet for tablet 910d440fa11b4f3982b1c95c772f474b (DEFAULT_TABLE table=test-workload [id=e6ea6b61341e4ad4aecd7688a407f508]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 21:25:39.202706 27142 tablet_service.cc:1467] Processing CreateTablet for tablet 5822df12a4f748d9ad2a4d27a527f689 (DEFAULT_TABLE table=test-workload [id=e6ea6b61341e4ad4aecd7688a407f508]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 21:25:39.202947 26986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 910d440fa11b4f3982b1c95c772f474b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.203212 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5822df12a4f748d9ad2a4d27a527f689. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.206461 27064 tablet_service.cc:1467] Processing CreateTablet for tablet d31826643a774c19b892b9ddcfac86a6 (DEFAULT_TABLE table=test-workload [id=e6ea6b61341e4ad4aecd7688a407f508]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 21:25:39.206928 27064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31826643a774c19b892b9ddcfac86a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.209167 27212 tablet_bootstrap.cc:492] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7: Bootstrap starting.
I20240502 21:25:39.210932 27212 tablet_bootstrap.cc:654] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.213011 27213 tablet_bootstrap.cc:492] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74: Bootstrap starting.
I20240502 21:25:39.213644 27215 tablet_bootstrap.cc:492] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d: Bootstrap starting.
I20240502 21:25:39.213856 27212 tablet_bootstrap.cc:492] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7: No bootstrap required, opened a new log
I20240502 21:25:39.213999 27212 ts_tablet_manager.cc:1367] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:25:39.214601 27212 raft_consensus.cc:348] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a040f981d7429cbfd7669406e803a7" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 40933 } }
I20240502 21:25:39.214763 27212 raft_consensus.cc:374] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.214859 27212 raft_consensus.cc:729] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a040f981d7429cbfd7669406e803a7, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.215037 27215 tablet_bootstrap.cc:654] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.215065 27212 consensus_queue.cc:260] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a040f981d7429cbfd7669406e803a7" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 40933 } }
I20240502 21:25:39.215257 27212 raft_consensus.cc:388] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:39.215342 27212 raft_consensus.cc:482] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:39.215453 27212 raft_consensus.cc:3037] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.216807 27215 tablet_bootstrap.cc:492] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d: No bootstrap required, opened a new log
I20240502 21:25:39.216949 27215 ts_tablet_manager.cc:1367] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:39.217574 27215 raft_consensus.cc:348] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d405a32679f4da486c0fd6362ff108d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 33535 } }
I20240502 21:25:39.217736 27215 raft_consensus.cc:374] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.217839 27215 raft_consensus.cc:729] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d405a32679f4da486c0fd6362ff108d, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.217931 27213 tablet_bootstrap.cc:654] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.217957 27212 raft_consensus.cc:504] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a040f981d7429cbfd7669406e803a7" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 40933 } }
I20240502 21:25:39.218245 27212 leader_election.cc:302] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15a040f981d7429cbfd7669406e803a7; no voters:
I20240502 21:25:39.218209 27215 consensus_queue.cc:260] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d405a32679f4da486c0fd6362ff108d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 33535 } }
I20240502 21:25:39.219435 27215 raft_consensus.cc:388] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:39.219550 27215 raft_consensus.cc:482] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:39.219671 27215 raft_consensus.cc:3037] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.221479 27215 raft_consensus.cc:504] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d405a32679f4da486c0fd6362ff108d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 33535 } }
I20240502 21:25:39.221689 27215 leader_election.cc:302] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d405a32679f4da486c0fd6362ff108d; no voters:
I20240502 21:25:39.222213 27215 leader_election.cc:288] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:39.222369 27219 raft_consensus.cc:2781] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:39.224793 27215 ts_tablet_manager.cc:1398] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:25:39.225672 27212 leader_election.cc:288] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:39.225816 27217 raft_consensus.cc:2781] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:39.226820 27219 raft_consensus.cc:686] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [term 1 LEADER]: Becoming Leader. State: Replica: 2d405a32679f4da486c0fd6362ff108d, State: Running, Role: LEADER
I20240502 21:25:39.227196 27219 consensus_queue.cc:237] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d405a32679f4da486c0fd6362ff108d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 33535 } }
I20240502 21:25:39.227303 27217 raft_consensus.cc:686] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [term 1 LEADER]: Becoming Leader. State: Replica: 15a040f981d7429cbfd7669406e803a7, State: Running, Role: LEADER
I20240502 21:25:39.227309 27213 tablet_bootstrap.cc:492] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74: No bootstrap required, opened a new log
I20240502 21:25:39.227793 27213 ts_tablet_manager.cc:1367] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20240502 21:25:39.227725 27217 consensus_queue.cc:237] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a040f981d7429cbfd7669406e803a7" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 40933 } }
I20240502 21:25:39.228721 27213 raft_consensus.cc:348] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d084f683b4274a438a4b1f8a4deddc74" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 42389 } }
I20240502 21:25:39.228964 27213 raft_consensus.cc:374] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.229108 27213 raft_consensus.cc:729] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d084f683b4274a438a4b1f8a4deddc74, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.229415 27213 consensus_queue.cc:260] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d084f683b4274a438a4b1f8a4deddc74" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 42389 } }
I20240502 21:25:39.229682 27213 raft_consensus.cc:388] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:39.229806 27213 raft_consensus.cc:482] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:39.229972 27213 raft_consensus.cc:3037] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.233017 27213 raft_consensus.cc:504] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d084f683b4274a438a4b1f8a4deddc74" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 42389 } }
I20240502 21:25:39.233343 27213 leader_election.cc:302] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d084f683b4274a438a4b1f8a4deddc74; no voters:
I20240502 21:25:39.234794 26859 catalog_manager.cc:5484] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15a040f981d7429cbfd7669406e803a7 (127.25.16.67). New cstate: current_term: 1 leader_uuid: "15a040f981d7429cbfd7669406e803a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a040f981d7429cbfd7669406e803a7" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 40933 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:39.240731 27226 raft_consensus.cc:2781] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:39.241153 27213 leader_election.cc:288] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:39.253495 27212 ts_tablet_manager.cc:1398] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7: Time spent starting tablet: real 0.039s user 0.010s sys 0.026s
I20240502 21:25:39.254731 27213 ts_tablet_manager.cc:1398] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74: Time spent starting tablet: real 0.027s user 0.008s sys 0.010s
I20240502 21:25:39.254987 27226 raft_consensus.cc:686] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [term 1 LEADER]: Becoming Leader. State: Replica: d084f683b4274a438a4b1f8a4deddc74, State: Running, Role: LEADER
I20240502 21:25:39.255689 27226 consensus_queue.cc:237] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d084f683b4274a438a4b1f8a4deddc74" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 42389 } }
I20240502 21:25:39.258512 26858 catalog_manager.cc:5484] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d405a32679f4da486c0fd6362ff108d (127.25.16.66). New cstate: current_term: 1 leader_uuid: "2d405a32679f4da486c0fd6362ff108d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d405a32679f4da486c0fd6362ff108d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 33535 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:39.270279 26859 catalog_manager.cc:5484] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 reported cstate change: term changed from 0 to 1, leader changed from <none> to d084f683b4274a438a4b1f8a4deddc74 (127.25.16.65). New cstate: current_term: 1 leader_uuid: "d084f683b4274a438a4b1f8a4deddc74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d084f683b4274a438a4b1f8a4deddc74" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 42389 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:39.294063 25665 master.cc:537] Master@127.25.16.124:42843 shutting down...
I20240502 21:25:39.300027 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:39.300349 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:39.300462 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P 22d8c50ce69045d099d53c704c434a84: stopping tablet replica
I20240502 21:25:39.305002 25665 master.cc:552] Master@127.25.16.124:42843 shutdown complete.
W20240502 21:25:40.266194 27177 heartbeater.cc:644] Failed to heartbeat to 127.25.16.124:42843 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.16.124:42843: connect: Connection refused (error 111)
I20240502 21:25:40.774781 27236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22d8c50ce69045d099d53c704c434a84)
I20240502 21:25:40.774971 27236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } }
I20240502 21:25:40.775555 27236 leader_election.cc:288] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66ea7a8ca11649e1a69ea248a450c8b1 (127.25.16.125:42649), 22d8c50ce69045d099d53c704c434a84 (127.25.16.124:42843)
I20240502 21:25:40.776224 26803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0102d924eb4b4051b3f4fafb85945f0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" is_pre_election: true
I20240502 21:25:40.776480 26803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0102d924eb4b4051b3f4fafb85945f0f in term 1.
I20240502 21:25:40.777156 26714 leader_election.cc:302] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0102d924eb4b4051b3f4fafb85945f0f, 66ea7a8ca11649e1a69ea248a450c8b1; no voters:
I20240502 21:25:40.777401 27236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:25:40.777508 27236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22d8c50ce69045d099d53c704c434a84)
I20240502 21:25:40.777591 27236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:25:40.777861 26714 leader_election.cc:334] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 22d8c50ce69045d099d53c704c434a84 (127.25.16.124:42843): Network error: Client connection negotiation failed: client connection to 127.25.16.124:42843: connect: Connection refused (error 111)
I20240502 21:25:40.780958 27236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } }
I20240502 21:25:40.781472 27236 leader_election.cc:288] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [CANDIDATE]: Term 2 election: Requested vote from peers 66ea7a8ca11649e1a69ea248a450c8b1 (127.25.16.125:42649), 22d8c50ce69045d099d53c704c434a84 (127.25.16.124:42843)
I20240502 21:25:40.781886 26803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0102d924eb4b4051b3f4fafb85945f0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "66ea7a8ca11649e1a69ea248a450c8b1"
I20240502 21:25:40.782084 26803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:25:40.782792 26714 leader_election.cc:334] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 22d8c50ce69045d099d53c704c434a84 (127.25.16.124:42843): Network error: Client connection negotiation failed: client connection to 127.25.16.124:42843: connect: Connection refused (error 111)
I20240502 21:25:40.783908 26803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0102d924eb4b4051b3f4fafb85945f0f in term 2.
I20240502 21:25:40.784237 26714 leader_election.cc:302] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0102d924eb4b4051b3f4fafb85945f0f, 66ea7a8ca11649e1a69ea248a450c8b1; no voters: 22d8c50ce69045d099d53c704c434a84
I20240502 21:25:40.784505 27236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:40.784826 27236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 2 LEADER]: Becoming Leader. State: Replica: 0102d924eb4b4051b3f4fafb85945f0f, State: Running, Role: LEADER
I20240502 21:25:40.785084 27236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } }
I20240502 21:25:40.786134 27239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0102d924eb4b4051b3f4fafb85945f0f. Latest consensus state: current_term: 2 leader_uuid: "0102d924eb4b4051b3f4fafb85945f0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0102d924eb4b4051b3f4fafb85945f0f" member_type: VOTER last_known_addr { host: "127.25.16.126" port: 41201 } } peers { permanent_uuid: "66ea7a8ca11649e1a69ea248a450c8b1" member_type: VOTER last_known_addr { host: "127.25.16.125" port: 42649 } } peers { permanent_uuid: "22d8c50ce69045d099d53c704c434a84" member_type: VOTER last_known_addr { host: "127.25.16.124" port: 42843 } } }
I20240502 21:25:40.786337 27239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:40.786784 27241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:40.788456 27241 catalog_manager.cc:669] Loaded metadata for table test-workload [id=e6ea6b61341e4ad4aecd7688a407f508]
I20240502 21:25:40.789752 27241 tablet_loader.cc:96] loaded metadata for tablet 5822df12a4f748d9ad2a4d27a527f689 (table test-workload [id=e6ea6b61341e4ad4aecd7688a407f508])
I20240502 21:25:40.790059 27241 tablet_loader.cc:96] loaded metadata for tablet 910d440fa11b4f3982b1c95c772f474b (table test-workload [id=e6ea6b61341e4ad4aecd7688a407f508])
I20240502 21:25:40.790311 27241 tablet_loader.cc:96] loaded metadata for tablet d31826643a774c19b892b9ddcfac86a6 (table test-workload [id=e6ea6b61341e4ad4aecd7688a407f508])
I20240502 21:25:40.790489 27241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:40.791239 27241 catalog_manager.cc:1259] Loaded cluster ID: 36ca7bf707fb4e7b9acac4bda6bbf37f
I20240502 21:25:40.791308 27241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:40.792392 27241 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:40.793170 27241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f: Loaded TSK: 0
I20240502 21:25:40.793756 27241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:40.888725 26926 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:40.888849 26926 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:40.945001 25665 tablet_server.cc:178] TabletServer@127.25.16.65:0 shutting down...
I20240502 21:25:40.952508 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:25:40.952780 25665 tablet_replica.cc:330] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74: stopping tablet replica
I20240502 21:25:40.952972 25665 raft_consensus.cc:2229] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:40.953162 25665 raft_consensus.cc:2258] T 910d440fa11b4f3982b1c95c772f474b P d084f683b4274a438a4b1f8a4deddc74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:40.966317 25665 tablet_server.cc:194] TabletServer@127.25.16.65:0 shutdown complete.
I20240502 21:25:40.969039 25665 tablet_server.cc:178] TabletServer@127.25.16.66:0 shutting down...
I20240502 21:25:40.976092 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:25:40.976328 25665 tablet_replica.cc:330] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d: stopping tablet replica
I20240502 21:25:40.976505 25665 raft_consensus.cc:2229] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:40.976671 25665 raft_consensus.cc:2258] T d31826643a774c19b892b9ddcfac86a6 P 2d405a32679f4da486c0fd6362ff108d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:40.989660 25665 tablet_server.cc:194] TabletServer@127.25.16.66:0 shutdown complete.
I20240502 21:25:40.992132 25665 tablet_server.cc:178] TabletServer@127.25.16.67:0 shutting down...
I20240502 21:25:40.999891 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:25:41.000121 25665 tablet_replica.cc:330] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7: stopping tablet replica
I20240502 21:25:41.000267 25665 raft_consensus.cc:2229] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:25:41.000451 25665 raft_consensus.cc:2258] T 5822df12a4f748d9ad2a4d27a527f689 P 15a040f981d7429cbfd7669406e803a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:41.013324 25665 tablet_server.cc:194] TabletServer@127.25.16.67:0 shutdown complete.
I20240502 21:25:41.015812 25665 master.cc:537] Master@127.25.16.126:41201 shutting down...
I20240502 21:25:41.022210 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 2 LEADER]: Raft consensus shutting down.
I20240502 21:25:41.022454 25665 pending_rounds.cc:62] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f: Trying to abort 1 pending ops.
I20240502 21:25:41.022598 25665 pending_rounds.cc:68] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f: Aborting op as it isn't in flight: id { term: 2 index: 11 } timestamp: 7023350336657682432 op_type: NO_OP noop_request { }
I20240502 21:25:41.022756 25665 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 21:25:41.022881 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:41.023047 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0102d924eb4b4051b3f4fafb85945f0f: stopping tablet replica
I20240502 21:25:41.036257 25665 master.cc:552] Master@127.25.16.126:41201 shutdown complete.
I20240502 21:25:41.040210 25665 master.cc:537] Master@127.25.16.125:42649 shutting down...
I20240502 21:25:41.046788 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:41.046977 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:41.047097 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P 66ea7a8ca11649e1a69ea248a450c8b1: stopping tablet replica
I20240502 21:25:41.059823 25665 master.cc:552] Master@127.25.16.125:42649 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (4350 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240502 21:25:41.070684 25665 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.16.126:33931
I20240502 21:25:41.071136 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:41.073267 27242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:41.073649 27243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:41.073844 27245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:41.074646 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:41.074904 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:41.074975 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:41.075026 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685141075010 us; error 0 us; skew 500 ppm
I20240502 21:25:41.075186 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:41.076381 25665 webserver.cc:460] Webserver started at http://127.25.16.126:44329/ using document root <none> and password file <none>
I20240502 21:25:41.076612 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:41.076684 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:41.076819 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:41.077401 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/master-0-root/instance:
uuid: "3e00760d1b724a11966281c3a05593ab"
format_stamp: "Formatted at 2024-05-02 21:25:41 on dist-test-slave-b524"
I20240502 21:25:41.079687 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:41.081218 27250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.081562 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 21:25:41.081653 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/master-0-root
uuid: "3e00760d1b724a11966281c3a05593ab"
format_stamp: "Formatted at 2024-05-02 21:25:41 on dist-test-slave-b524"
I20240502 21:25:41.081756 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:41.091017 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:41.091482 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:41.103062 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.126:33931
I20240502 21:25:41.103081 27301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.126:33931 every 8 connection(s)
I20240502 21:25:41.104246 27302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.107988 27302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab: Bootstrap starting.
I20240502 21:25:41.109194 27302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.110586 27302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab: No bootstrap required, opened a new log
I20240502 21:25:41.111150 27302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e00760d1b724a11966281c3a05593ab" member_type: VOTER }
I20240502 21:25:41.111284 27302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.111373 27302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e00760d1b724a11966281c3a05593ab, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.111558 27302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e00760d1b724a11966281c3a05593ab" member_type: VOTER }
I20240502 21:25:41.111707 27302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:41.111775 27302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:41.111887 27302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.113517 27302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e00760d1b724a11966281c3a05593ab" member_type: VOTER }
I20240502 21:25:41.113723 27302 leader_election.cc:302] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e00760d1b724a11966281c3a05593ab; no voters:
I20240502 21:25:41.114058 27302 leader_election.cc:288] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:41.114198 27305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:41.114619 27305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [term 1 LEADER]: Becoming Leader. State: Replica: 3e00760d1b724a11966281c3a05593ab, State: Running, Role: LEADER
I20240502 21:25:41.114866 27302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:41.114884 27305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e00760d1b724a11966281c3a05593ab" member_type: VOTER }
I20240502 21:25:41.115746 27306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e00760d1b724a11966281c3a05593ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e00760d1b724a11966281c3a05593ab" member_type: VOTER } }
I20240502 21:25:41.115831 27307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e00760d1b724a11966281c3a05593ab. Latest consensus state: current_term: 1 leader_uuid: "3e00760d1b724a11966281c3a05593ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e00760d1b724a11966281c3a05593ab" member_type: VOTER } }
I20240502 21:25:41.115999 27306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:41.116127 27307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:41.116933 27310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:41.118297 27310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:41.119997 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:25:41.121338 27310 catalog_manager.cc:1347] Generated new cluster ID: 8bee656f7a0f42429d8b3bcadd7a07fb
I20240502 21:25:41.121444 27310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:41.129156 27310 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:41.129853 27310 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:41.137445 27310 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab: Generated new TSK 0
I20240502 21:25:41.137645 27310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:41.152534 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:41.154862 27323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:41.155128 27324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:41.155376 27326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:41.155814 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:41.156039 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:41.156095 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:41.156144 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685141156120 us; error 0 us; skew 500 ppm
I20240502 21:25:41.156265 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:41.157469 25665 webserver.cc:460] Webserver started at http://127.25.16.65:46549/ using document root <none> and password file <none>
I20240502 21:25:41.157663 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:41.157737 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:41.157835 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:41.158382 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-0-root/instance:
uuid: "04044e2347154e9fbfdc903e712c1b6b"
format_stamp: "Formatted at 2024-05-02 21:25:41 on dist-test-slave-b524"
I20240502 21:25:41.160612 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:41.162081 27331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.162400 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:25:41.162503 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-0-root
uuid: "04044e2347154e9fbfdc903e712c1b6b"
format_stamp: "Formatted at 2024-05-02 21:25:41 on dist-test-slave-b524"
I20240502 21:25:41.162639 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:41.172792 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:41.173257 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:41.173745 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:41.174521 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:41.174588 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.174691 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:41.174749 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.187296 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.65:39895
I20240502 21:25:41.187317 27393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.65:39895 every 8 connection(s)
I20240502 21:25:41.188824 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:41.191687 27398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:41.191730 27399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:41.193733 27401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:41.194109 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:41.194366 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:41.194443 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:41.194509 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685141194482 us; error 0 us; skew 500 ppm
I20240502 21:25:41.194684 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:41.196024 27394 heartbeater.cc:347] Connected to a master server at 127.25.16.126:33931
I20240502 21:25:41.196162 27394 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:41.196604 27394 heartbeater.cc:511] Master 127.25.16.126:33931 requested a full tablet report, sending...
I20240502 21:25:41.197465 27267 ts_manager.cc:194] Registered new tserver with Master: 04044e2347154e9fbfdc903e712c1b6b (127.25.16.65:39895)
I20240502 21:25:41.198977 27267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58654
I20240502 21:25:41.205531 25665 webserver.cc:460] Webserver started at http://127.25.16.66:40817/ using document root <none> and password file <none>
I20240502 21:25:41.205720 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:41.205791 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:41.205899 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:41.206398 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-1-root/instance:
uuid: "893e15fd8bec45faa1f7f499d07dbb12"
format_stamp: "Formatted at 2024-05-02 21:25:41 on dist-test-slave-b524"
I20240502 21:25:41.208415 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:25:41.209646 27406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.209919 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:25:41.210000 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-1-root
uuid: "893e15fd8bec45faa1f7f499d07dbb12"
format_stamp: "Formatted at 2024-05-02 21:25:41 on dist-test-slave-b524"
I20240502 21:25:41.210098 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:41.224686 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:41.225086 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:41.225502 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:41.226168 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:41.226218 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.226274 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:41.226312 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.237864 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.66:37299
I20240502 21:25:41.237886 27469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.66:37299 every 8 connection(s)
I20240502 21:25:41.239190 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:41.241448 27473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:41.241763 27474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:41.242941 27477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:41.245229 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:41.245483 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:41.245539 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:41.245568 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685141245565 us; error 0 us; skew 500 ppm
I20240502 21:25:41.245690 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:41.245963 27470 heartbeater.cc:347] Connected to a master server at 127.25.16.126:33931
I20240502 21:25:41.246071 27470 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:41.246395 27470 heartbeater.cc:511] Master 127.25.16.126:33931 requested a full tablet report, sending...
I20240502 21:25:41.246903 25665 webserver.cc:460] Webserver started at http://127.25.16.67:36653/ using document root <none> and password file <none>
I20240502 21:25:41.247093 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:41.247134 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:41.247201 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:41.247205 27267 ts_manager.cc:194] Registered new tserver with Master: 893e15fd8bec45faa1f7f499d07dbb12 (127.25.16.66:37299)
I20240502 21:25:41.247866 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-2-root/instance:
uuid: "74469025e94a4fc3b03b5a51763366d6"
format_stamp: "Formatted at 2024-05-02 21:25:41 on dist-test-slave-b524"
I20240502 21:25:41.248214 27267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58658
I20240502 21:25:41.251210 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:41.252509 27481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.252789 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:25:41.252875 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-2-root
uuid: "74469025e94a4fc3b03b5a51763366d6"
format_stamp: "Formatted at 2024-05-02 21:25:41 on dist-test-slave-b524"
I20240502 21:25:41.252972 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:41.272135 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:41.272604 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:41.273068 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:41.273754 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:41.273808 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.273867 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:41.273900 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.285326 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.67:35109
I20240502 21:25:41.285359 27544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.67:35109 every 8 connection(s)
I20240502 21:25:41.291007 27545 heartbeater.cc:347] Connected to a master server at 127.25.16.126:33931
I20240502 21:25:41.291112 27545 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:41.291414 27545 heartbeater.cc:511] Master 127.25.16.126:33931 requested a full tablet report, sending...
I20240502 21:25:41.292199 27267 ts_manager.cc:194] Registered new tserver with Master: 74469025e94a4fc3b03b5a51763366d6 (127.25.16.67:35109)
I20240502 21:25:41.292891 27267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58664
I20240502 21:25:41.292932 25665 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006772791s
I20240502 21:25:42.200269 27394 heartbeater.cc:503] Master 127.25.16.126:33931 was elected leader, sending a full tablet report...
I20240502 21:25:42.250223 27470 heartbeater.cc:503] Master 127.25.16.126:33931 was elected leader, sending a full tablet report...
I20240502 21:25:42.294021 27545 heartbeater.cc:503] Master 127.25.16.126:33931 was elected leader, sending a full tablet report...
I20240502 21:25:42.319594 25665 test_util.cc:268] Using random seed: 1053770185
I20240502 21:25:42.330057 27267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38662:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 21:25:42.330651 27267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:25:42.347965 27359 tablet_service.cc:1467] Processing CreateTablet for tablet 94a4ce9c29194904884136342aeb5289 (DEFAULT_TABLE table=test-workload [id=806a8d74f99d4fd68a4c7f4999a49382]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 21:25:42.348106 27358 tablet_service.cc:1467] Processing CreateTablet for tablet bdbac986d7b441818fc3b3038b76bb3e (DEFAULT_TABLE table=test-workload [id=806a8d74f99d4fd68a4c7f4999a49382]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 21:25:42.348520 27359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a4ce9c29194904884136342aeb5289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.348737 27358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdbac986d7b441818fc3b3038b76bb3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.349231 27509 tablet_service.cc:1467] Processing CreateTablet for tablet bdbac986d7b441818fc3b3038b76bb3e (DEFAULT_TABLE table=test-workload [id=806a8d74f99d4fd68a4c7f4999a49382]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 21:25:42.349694 27509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdbac986d7b441818fc3b3038b76bb3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.352916 27510 tablet_service.cc:1467] Processing CreateTablet for tablet 94a4ce9c29194904884136342aeb5289 (DEFAULT_TABLE table=test-workload [id=806a8d74f99d4fd68a4c7f4999a49382]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 21:25:42.353320 27510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a4ce9c29194904884136342aeb5289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.355875 27434 tablet_service.cc:1467] Processing CreateTablet for tablet bdbac986d7b441818fc3b3038b76bb3e (DEFAULT_TABLE table=test-workload [id=806a8d74f99d4fd68a4c7f4999a49382]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 21:25:42.356268 27434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdbac986d7b441818fc3b3038b76bb3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.356564 27565 tablet_bootstrap.cc:492] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b: Bootstrap starting.
I20240502 21:25:42.356782 27435 tablet_service.cc:1467] Processing CreateTablet for tablet 94a4ce9c29194904884136342aeb5289 (DEFAULT_TABLE table=test-workload [id=806a8d74f99d4fd68a4c7f4999a49382]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 21:25:42.357240 27435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a4ce9c29194904884136342aeb5289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.359524 27566 tablet_bootstrap.cc:492] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6: Bootstrap starting.
I20240502 21:25:42.360610 27565 tablet_bootstrap.cc:654] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.361346 27566 tablet_bootstrap.cc:654] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.365578 27569 tablet_bootstrap.cc:492] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12: Bootstrap starting.
I20240502 21:25:42.365751 27565 tablet_bootstrap.cc:492] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b: No bootstrap required, opened a new log
I20240502 21:25:42.365908 27565 ts_tablet_manager.cc:1367] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240502 21:25:42.366662 27565 raft_consensus.cc:348] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } }
I20240502 21:25:42.367357 27569 tablet_bootstrap.cc:654] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.367444 27565 raft_consensus.cc:374] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.367587 27565 raft_consensus.cc:729] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04044e2347154e9fbfdc903e712c1b6b, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.367892 27565 consensus_queue.cc:260] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } }
I20240502 21:25:42.368085 27566 tablet_bootstrap.cc:492] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6: No bootstrap required, opened a new log
I20240502 21:25:42.368191 27566 ts_tablet_manager.cc:1367] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240502 21:25:42.368799 27565 ts_tablet_manager.cc:1398] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:25:42.368963 27566 raft_consensus.cc:348] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } }
I20240502 21:25:42.369097 27565 tablet_bootstrap.cc:492] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b: Bootstrap starting.
I20240502 21:25:42.369216 27566 raft_consensus.cc:374] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.369378 27566 raft_consensus.cc:729] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74469025e94a4fc3b03b5a51763366d6, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.369661 27566 consensus_queue.cc:260] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } }
I20240502 21:25:42.370816 27565 tablet_bootstrap.cc:654] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.373673 27565 tablet_bootstrap.cc:492] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b: No bootstrap required, opened a new log
I20240502 21:25:42.373726 27566 ts_tablet_manager.cc:1398] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 21:25:42.373800 27565 ts_tablet_manager.cc:1367] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240502 21:25:42.373994 27566 tablet_bootstrap.cc:492] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6: Bootstrap starting.
I20240502 21:25:42.374487 27569 tablet_bootstrap.cc:492] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12: No bootstrap required, opened a new log
I20240502 21:25:42.374496 27565 raft_consensus.cc:348] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } }
I20240502 21:25:42.374760 27565 raft_consensus.cc:374] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.374831 27565 raft_consensus.cc:729] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04044e2347154e9fbfdc903e712c1b6b, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.374697 27569 ts_tablet_manager.cc:1367] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20240502 21:25:42.375036 27565 consensus_queue.cc:260] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } }
I20240502 21:25:42.375484 27566 tablet_bootstrap.cc:654] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.375933 27569 raft_consensus.cc:348] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } }
I20240502 21:25:42.376165 27569 raft_consensus.cc:374] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.376263 27569 raft_consensus.cc:729] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893e15fd8bec45faa1f7f499d07dbb12, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.376317 27565 ts_tablet_manager.cc:1398] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:25:42.376605 27569 consensus_queue.cc:260] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } }
I20240502 21:25:42.377719 27569 ts_tablet_manager.cc:1398] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:25:42.378114 27569 tablet_bootstrap.cc:492] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12: Bootstrap starting.
I20240502 21:25:42.379078 27566 tablet_bootstrap.cc:492] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6: No bootstrap required, opened a new log
I20240502 21:25:42.379201 27566 ts_tablet_manager.cc:1367] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:25:42.379609 27569 tablet_bootstrap.cc:654] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.379695 27566 raft_consensus.cc:348] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } }
I20240502 21:25:42.379901 27566 raft_consensus.cc:374] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.379961 27566 raft_consensus.cc:729] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74469025e94a4fc3b03b5a51763366d6, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.380175 27566 consensus_queue.cc:260] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } }
I20240502 21:25:42.380662 27566 ts_tablet_manager.cc:1398] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:25:42.385083 27569 tablet_bootstrap.cc:492] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12: No bootstrap required, opened a new log
I20240502 21:25:42.385179 27569 ts_tablet_manager.cc:1367] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240502 21:25:42.385624 27569 raft_consensus.cc:348] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } }
I20240502 21:25:42.385753 27569 raft_consensus.cc:374] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.385790 27569 raft_consensus.cc:729] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893e15fd8bec45faa1f7f499d07dbb12, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.385941 27569 consensus_queue.cc:260] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } }
I20240502 21:25:42.386302 27569 ts_tablet_manager.cc:1398] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:25:42.446080 27571 raft_consensus.cc:482] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:42.446223 27571 raft_consensus.cc:504] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } }
I20240502 21:25:42.446924 27571 leader_election.cc:288] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74469025e94a4fc3b03b5a51763366d6 (127.25.16.67:35109), 893e15fd8bec45faa1f7f499d07dbb12 (127.25.16.66:37299)
I20240502 21:25:42.453348 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a4ce9c29194904884136342aeb5289" candidate_uuid: "04044e2347154e9fbfdc903e712c1b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74469025e94a4fc3b03b5a51763366d6" is_pre_election: true
I20240502 21:25:42.453575 27520 raft_consensus.cc:2454] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04044e2347154e9fbfdc903e712c1b6b in term 0.
I20240502 21:25:42.453549 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a4ce9c29194904884136342aeb5289" candidate_uuid: "04044e2347154e9fbfdc903e712c1b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893e15fd8bec45faa1f7f499d07dbb12" is_pre_election: true
I20240502 21:25:42.453877 27445 raft_consensus.cc:2454] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04044e2347154e9fbfdc903e712c1b6b in term 0.
I20240502 21:25:42.454008 27335 leader_election.cc:302] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04044e2347154e9fbfdc903e712c1b6b, 74469025e94a4fc3b03b5a51763366d6; no voters:
I20240502 21:25:42.454255 27571 raft_consensus.cc:2781] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:42.454342 27571 raft_consensus.cc:482] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:42.454404 27571 raft_consensus.cc:3037] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:42.456239 27571 raft_consensus.cc:504] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } }
I20240502 21:25:42.456745 27571 leader_election.cc:288] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [CANDIDATE]: Term 1 election: Requested vote from peers 74469025e94a4fc3b03b5a51763366d6 (127.25.16.67:35109), 893e15fd8bec45faa1f7f499d07dbb12 (127.25.16.66:37299)
I20240502 21:25:42.457177 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a4ce9c29194904884136342aeb5289" candidate_uuid: "04044e2347154e9fbfdc903e712c1b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893e15fd8bec45faa1f7f499d07dbb12"
I20240502 21:25:42.457177 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a4ce9c29194904884136342aeb5289" candidate_uuid: "04044e2347154e9fbfdc903e712c1b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74469025e94a4fc3b03b5a51763366d6"
I20240502 21:25:42.457372 27445 raft_consensus.cc:3037] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:42.457405 27520 raft_consensus.cc:3037] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:42.460103 27520 raft_consensus.cc:2454] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04044e2347154e9fbfdc903e712c1b6b in term 1.
I20240502 21:25:42.460103 27445 raft_consensus.cc:2454] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04044e2347154e9fbfdc903e712c1b6b in term 1.
I20240502 21:25:42.460494 27335 leader_election.cc:302] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04044e2347154e9fbfdc903e712c1b6b, 74469025e94a4fc3b03b5a51763366d6; no voters:
I20240502 21:25:42.460702 27571 raft_consensus.cc:2781] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:42.461055 27571 raft_consensus.cc:686] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 1 LEADER]: Becoming Leader. State: Replica: 04044e2347154e9fbfdc903e712c1b6b, State: Running, Role: LEADER
I20240502 21:25:42.461285 27571 consensus_queue.cc:237] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } }
I20240502 21:25:42.463802 27266 catalog_manager.cc:5484] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 04044e2347154e9fbfdc903e712c1b6b (127.25.16.65). New cstate: current_term: 1 leader_uuid: "04044e2347154e9fbfdc903e712c1b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:42.626956 27575 raft_consensus.cc:482] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:42.627144 27575 raft_consensus.cc:504] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } }
I20240502 21:25:42.627969 27575 leader_election.cc:288] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04044e2347154e9fbfdc903e712c1b6b (127.25.16.65:39895), 74469025e94a4fc3b03b5a51763366d6 (127.25.16.67:35109)
I20240502 21:25:42.634268 27369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdbac986d7b441818fc3b3038b76bb3e" candidate_uuid: "893e15fd8bec45faa1f7f499d07dbb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04044e2347154e9fbfdc903e712c1b6b" is_pre_election: true
I20240502 21:25:42.634291 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdbac986d7b441818fc3b3038b76bb3e" candidate_uuid: "893e15fd8bec45faa1f7f499d07dbb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74469025e94a4fc3b03b5a51763366d6" is_pre_election: true
I20240502 21:25:42.634505 27369 raft_consensus.cc:2454] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 893e15fd8bec45faa1f7f499d07dbb12 in term 0.
I20240502 21:25:42.634523 27520 raft_consensus.cc:2454] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 893e15fd8bec45faa1f7f499d07dbb12 in term 0.
I20240502 21:25:42.634871 27410 leader_election.cc:302] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74469025e94a4fc3b03b5a51763366d6, 893e15fd8bec45faa1f7f499d07dbb12; no voters:
I20240502 21:25:42.635088 27575 raft_consensus.cc:2781] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:42.635167 27575 raft_consensus.cc:482] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:42.635231 27575 raft_consensus.cc:3037] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:42.636974 27575 raft_consensus.cc:504] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } }
I20240502 21:25:42.637413 27575 leader_election.cc:288] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [CANDIDATE]: Term 1 election: Requested vote from peers 04044e2347154e9fbfdc903e712c1b6b (127.25.16.65:39895), 74469025e94a4fc3b03b5a51763366d6 (127.25.16.67:35109)
I20240502 21:25:42.637763 27369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdbac986d7b441818fc3b3038b76bb3e" candidate_uuid: "893e15fd8bec45faa1f7f499d07dbb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04044e2347154e9fbfdc903e712c1b6b"
I20240502 21:25:42.637919 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdbac986d7b441818fc3b3038b76bb3e" candidate_uuid: "893e15fd8bec45faa1f7f499d07dbb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74469025e94a4fc3b03b5a51763366d6"
I20240502 21:25:42.637966 27369 raft_consensus.cc:3037] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:42.638065 27520 raft_consensus.cc:3037] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:42.640743 27520 raft_consensus.cc:2454] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 893e15fd8bec45faa1f7f499d07dbb12 in term 1.
I20240502 21:25:42.640751 27369 raft_consensus.cc:2454] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 893e15fd8bec45faa1f7f499d07dbb12 in term 1.
I20240502 21:25:42.641142 27407 leader_election.cc:302] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04044e2347154e9fbfdc903e712c1b6b, 74469025e94a4fc3b03b5a51763366d6, 893e15fd8bec45faa1f7f499d07dbb12; no voters:
I20240502 21:25:42.641412 27575 raft_consensus.cc:2781] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:42.641770 27575 raft_consensus.cc:686] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 1 LEADER]: Becoming Leader. State: Replica: 893e15fd8bec45faa1f7f499d07dbb12, State: Running, Role: LEADER
I20240502 21:25:42.642056 27575 consensus_queue.cc:237] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } }
I20240502 21:25:42.644271 27266 catalog_manager.cc:5484] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 893e15fd8bec45faa1f7f499d07dbb12 (127.25.16.66). New cstate: current_term: 1 leader_uuid: "893e15fd8bec45faa1f7f499d07dbb12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:42.923861 27571 consensus_queue.cc:1035] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:42.929075 27571 consensus_queue.cc:1035] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:43.118423 27575 consensus_queue.cc:1035] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:43.120100 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:43.120862 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:25:43.121042 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:43.123090 27584 consensus_queue.cc:1035] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:45.121388 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:45.122192 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:25:45.122356 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:47.122807 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:47.123373 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:25:47.123512 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:49.123891 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:49.124524 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:25:49.124653 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:51.125062 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:51.125643 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:25:51.125783 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:53.126155 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:53.126760 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:25:53.126879 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:53.756496 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:53.759517 27597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:53.760133 27598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:53.761078 27600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:53.761858 25665 server_base.cc:1034] running on GCE node
I20240502 21:25:53.762087 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:25:53.762156 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:25:53.762197 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685153762184 us; error 0 us; skew 500 ppm
I20240502 21:25:53.762323 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:53.763568 25665 webserver.cc:460] Webserver started at http://127.25.16.68:45701/ using document root <none> and password file <none>
I20240502 21:25:53.763828 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:53.763902 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:53.764045 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:53.764722 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-3-root/instance:
uuid: "07556ecb8098420883d5b64be2f18611"
format_stamp: "Formatted at 2024-05-02 21:25:53 on dist-test-slave-b524"
I20240502 21:25:53.767094 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:53.768585 27605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:53.768952 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 21:25:53.769048 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-3-root
uuid: "07556ecb8098420883d5b64be2f18611"
format_stamp: "Formatted at 2024-05-02 21:25:53 on dist-test-slave-b524"
I20240502 21:25:53.769150 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:53.793130 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:53.793617 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:53.794095 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:53.794829 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:53.794905 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:53.794970 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:53.795022 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:53.807746 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.68:33559
I20240502 21:25:53.807798 27668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.68:33559 every 8 connection(s)
I20240502 21:25:53.814363 27669 heartbeater.cc:347] Connected to a master server at 127.25.16.126:33931
I20240502 21:25:53.814491 27669 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:53.814913 27669 heartbeater.cc:511] Master 127.25.16.126:33931 requested a full tablet report, sending...
I20240502 21:25:53.815745 27266 ts_manager.cc:194] Registered new tserver with Master: 07556ecb8098420883d5b64be2f18611 (127.25.16.68:33559)
I20240502 21:25:53.816534 27266 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45864
I20240502 21:25:54.141836 27445 consensus_queue.cc:237] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: true } }
I20240502 21:25:54.144336 27369 raft_consensus.cc:1261] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 1 FOLLOWER]: Refusing update from remote peer 893e15fd8bec45faa1f7f499d07dbb12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:54.144837 27520 raft_consensus.cc:1261] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [term 1 FOLLOWER]: Refusing update from remote peer 893e15fd8bec45faa1f7f499d07dbb12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:54.145027 27677 consensus_queue.cc:1035] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:54.145439 27595 consensus_queue.cc:1035] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:54.148402 27595 raft_consensus.cc:2932] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 07556ecb8098420883d5b64be2f18611 (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: true } } }
I20240502 21:25:54.149655 27520 raft_consensus.cc:2932] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 07556ecb8098420883d5b64be2f18611 (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: true } } }
I20240502 21:25:54.151053 27368 raft_consensus.cc:2932] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 07556ecb8098420883d5b64be2f18611 (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: true } } }
W20240502 21:25:54.153506 27410 consensus_peers.cc:485] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 -> Peer 07556ecb8098420883d5b64be2f18611 (127.25.16.68:33559): Couldn't send request to peer 07556ecb8098420883d5b64be2f18611. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bdbac986d7b441818fc3b3038b76bb3e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:54.153961 27266 catalog_manager.cc:5484] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 reported cstate change: config changed from index -1 to 2, NON_VOTER 07556ecb8098420883d5b64be2f18611 (127.25.16.68) added. New cstate: current_term: 1 leader_uuid: "893e15fd8bec45faa1f7f499d07dbb12" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:54.623917 27687 ts_tablet_manager.cc:902] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611: Initiating tablet copy from peer 893e15fd8bec45faa1f7f499d07dbb12 (127.25.16.66:37299)
I20240502 21:25:54.625550 27687 tablet_copy_client.cc:308] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.66:37299
I20240502 21:25:54.631731 27455 tablet_copy_service.cc:140] P 893e15fd8bec45faa1f7f499d07dbb12: Received BeginTabletCopySession request for tablet bdbac986d7b441818fc3b3038b76bb3e from peer 07556ecb8098420883d5b64be2f18611 ({username='slave'} at 127.0.0.1:34018)
I20240502 21:25:54.631909 27455 tablet_copy_service.cc:161] P 893e15fd8bec45faa1f7f499d07dbb12: Beginning new tablet copy session on tablet bdbac986d7b441818fc3b3038b76bb3e from peer 07556ecb8098420883d5b64be2f18611 at {username='slave'} at 127.0.0.1:34018: session id = 07556ecb8098420883d5b64be2f18611-bdbac986d7b441818fc3b3038b76bb3e
I20240502 21:25:54.635007 27455 tablet_copy_source_session.cc:205] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:54.636804 27687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdbac986d7b441818fc3b3038b76bb3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:54.642673 27687 tablet_copy_client.cc:783] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:54.642917 27687 tablet_copy_client.cc:647] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:54.644757 27687 tablet_copy_client.cc:523] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:54.648008 27687 tablet_bootstrap.cc:492] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611: Bootstrap starting.
I20240502 21:25:54.654481 27687 tablet_bootstrap.cc:492] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:54.654788 27687 tablet_bootstrap.cc:492] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611: Bootstrap complete.
I20240502 21:25:54.654983 27687 ts_tablet_manager.cc:1367] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20240502 21:25:54.655596 27687 raft_consensus.cc:348] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: true } }
I20240502 21:25:54.655820 27687 raft_consensus.cc:729] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 07556ecb8098420883d5b64be2f18611, State: Initialized, Role: LEARNER
I20240502 21:25:54.655997 27687 consensus_queue.cc:260] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: true } }
I20240502 21:25:54.656680 27669 heartbeater.cc:503] Master 127.25.16.126:33931 was elected leader, sending a full tablet report...
I20240502 21:25:54.656762 27687 ts_tablet_manager.cc:1398] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240502 21:25:54.657438 27455 tablet_copy_service.cc:342] P 893e15fd8bec45faa1f7f499d07dbb12: Request end of tablet copy session 07556ecb8098420883d5b64be2f18611-bdbac986d7b441818fc3b3038b76bb3e received from {username='slave'} at 127.0.0.1:34018
I20240502 21:25:54.657552 27455 tablet_copy_service.cc:433] P 893e15fd8bec45faa1f7f499d07dbb12: ending tablet copy session 07556ecb8098420883d5b64be2f18611-bdbac986d7b441818fc3b3038b76bb3e on tablet bdbac986d7b441818fc3b3038b76bb3e with peer 07556ecb8098420883d5b64be2f18611
I20240502 21:25:55.127252 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:55.127874 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:25:55.128021 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:55.215348 27643 raft_consensus.cc:1203] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:25:55.755438 27695 raft_consensus.cc:1050] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12: attempting to promote NON_VOTER 07556ecb8098420883d5b64be2f18611 to VOTER
I20240502 21:25:55.756312 27695 consensus_queue.cc:237] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false } }
I20240502 21:25:55.758525 27520 raft_consensus.cc:1261] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [term 1 FOLLOWER]: Refusing update from remote peer 893e15fd8bec45faa1f7f499d07dbb12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:25:55.758554 27643 raft_consensus.cc:1261] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611 [term 1 LEARNER]: Refusing update from remote peer 893e15fd8bec45faa1f7f499d07dbb12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:25:55.759124 27698 consensus_queue.cc:1035] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:25:55.759647 27698 consensus_queue.cc:1035] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:25:55.762718 27698 raft_consensus.cc:2932] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 07556ecb8098420883d5b64be2f18611 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false } } }
I20240502 21:25:55.763073 27643 raft_consensus.cc:2932] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 07556ecb8098420883d5b64be2f18611 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false } } }
I20240502 21:25:55.768158 27267 catalog_manager.cc:5484] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611 reported cstate change: config changed from index 2 to 3, 07556ecb8098420883d5b64be2f18611 (127.25.16.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "893e15fd8bec45faa1f7f499d07dbb12" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false } } }
I20240502 21:25:55.770701 27368 raft_consensus.cc:1261] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 1 FOLLOWER]: Refusing update from remote peer 893e15fd8bec45faa1f7f499d07dbb12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:25:55.770712 27520 raft_consensus.cc:2932] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 07556ecb8098420883d5b64be2f18611 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false } } }
I20240502 21:25:55.774176 27706 consensus_queue.cc:1035] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:25:55.775671 27368 raft_consensus.cc:2932] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 07556ecb8098420883d5b64be2f18611 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "74469025e94a4fc3b03b5a51763366d6" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 35109 } attrs { replace: true } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false } } }
I20240502 21:25:56.335990 27445 consensus_queue.cc:237] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false } }
I20240502 21:25:56.337692 27643 raft_consensus.cc:1261] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611 [term 1 FOLLOWER]: Refusing update from remote peer 893e15fd8bec45faa1f7f499d07dbb12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:25:56.337709 27368 raft_consensus.cc:1261] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 1 FOLLOWER]: Refusing update from remote peer 893e15fd8bec45faa1f7f499d07dbb12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:25:56.338228 27713 consensus_queue.cc:1035] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:25:56.338773 27712 consensus_queue.cc:1035] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:25:56.340471 27706 raft_consensus.cc:2932] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74469025e94a4fc3b03b5a51763366d6 (127.25.16.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false } } }
I20240502 21:25:56.341583 27643 raft_consensus.cc:2932] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74469025e94a4fc3b03b5a51763366d6 (127.25.16.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false } } }
I20240502 21:25:56.343775 27368 raft_consensus.cc:2932] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74469025e94a4fc3b03b5a51763366d6 (127.25.16.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false } } }
I20240502 21:25:56.344597 27265 catalog_manager.cc:5484] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 reported cstate change: config changed from index 3 to 4, VOTER 74469025e94a4fc3b03b5a51763366d6 (127.25.16.67) evicted. New cstate: current_term: 1 leader_uuid: "893e15fd8bec45faa1f7f499d07dbb12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04044e2347154e9fbfdc903e712c1b6b" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 39895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "893e15fd8bec45faa1f7f499d07dbb12" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 37299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07556ecb8098420883d5b64be2f18611" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 33559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:56.346987 27254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bdbac986d7b441818fc3b3038b76bb3e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:25:56.349581 27510 tablet_service.cc:1514] Processing DeleteTablet for tablet bdbac986d7b441818fc3b3038b76bb3e with delete_type TABLET_DATA_TOMBSTONED (TS 74469025e94a4fc3b03b5a51763366d6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54196
I20240502 21:25:56.350507 27718 tablet_replica.cc:330] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6: stopping tablet replica
I20240502 21:25:56.350693 27718 raft_consensus.cc:2229] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:56.351012 27718 raft_consensus.cc:2258] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:56.351775 27718 ts_tablet_manager.cc:1875] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:25:56.355971 27718 ts_tablet_manager.cc:1888] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:25:56.356074 27718 log.cc:1196] T bdbac986d7b441818fc3b3038b76bb3e P 74469025e94a4fc3b03b5a51763366d6: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714685126025081-25665-0/minicluster-data/ts-2-root/wals/bdbac986d7b441818fc3b3038b76bb3e
I20240502 21:25:56.356644 27254 catalog_manager.cc:4830] TS 74469025e94a4fc3b03b5a51763366d6 (127.25.16.67:35109): tablet bdbac986d7b441818fc3b3038b76bb3e (table test-workload [id=806a8d74f99d4fd68a4c7f4999a49382]) successfully deleted
I20240502 21:25:57.128451 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:57.129086 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:25:57.129218 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:25:59.129598 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:25:59.130239 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:25:59.130369 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:01.130869 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:01.131520 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:01.131671 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:03.132093 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:03.132782 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:03.132923 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:05.133368 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:05.134043 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:05.134178 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:07.134625 27320 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:07.135265 27320 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:07.135401 27320 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:07.894840 25665 tablet_server.cc:178] TabletServer@127.25.16.65:0 shutting down...
I20240502 21:26:07.905977 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:07.906236 25665 tablet_replica.cc:330] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b: stopping tablet replica
I20240502 21:26:07.906414 25665 raft_consensus.cc:2229] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:07.906745 25665 raft_consensus.cc:2258] T 94a4ce9c29194904884136342aeb5289 P 04044e2347154e9fbfdc903e712c1b6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:07.907364 25665 tablet_replica.cc:330] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b: stopping tablet replica
I20240502 21:26:07.907510 25665 raft_consensus.cc:2229] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:07.907722 25665 raft_consensus.cc:2258] T bdbac986d7b441818fc3b3038b76bb3e P 04044e2347154e9fbfdc903e712c1b6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:07.921548 25665 tablet_server.cc:194] TabletServer@127.25.16.65:0 shutdown complete.
I20240502 21:26:07.924595 25665 tablet_server.cc:178] TabletServer@127.25.16.66:0 shutting down...
I20240502 21:26:07.940060 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:07.940351 25665 tablet_replica.cc:330] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12: stopping tablet replica
I20240502 21:26:07.940537 25665 raft_consensus.cc:2229] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:07.940740 25665 raft_consensus.cc:2258] T 94a4ce9c29194904884136342aeb5289 P 893e15fd8bec45faa1f7f499d07dbb12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:07.941355 25665 tablet_replica.cc:330] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12: stopping tablet replica
I20240502 21:26:07.941488 25665 raft_consensus.cc:2229] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:07.941782 25665 raft_consensus.cc:2258] T bdbac986d7b441818fc3b3038b76bb3e P 893e15fd8bec45faa1f7f499d07dbb12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:07.955670 25665 tablet_server.cc:194] TabletServer@127.25.16.66:0 shutdown complete.
I20240502 21:26:07.958976 25665 tablet_server.cc:178] TabletServer@127.25.16.67:0 shutting down...
I20240502 21:26:07.966660 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:07.966939 25665 tablet_replica.cc:330] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6: stopping tablet replica
I20240502 21:26:07.967115 25665 raft_consensus.cc:2229] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:07.967295 25665 raft_consensus.cc:2258] T 94a4ce9c29194904884136342aeb5289 P 74469025e94a4fc3b03b5a51763366d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:07.981072 25665 tablet_server.cc:194] TabletServer@127.25.16.67:0 shutdown complete.
I20240502 21:26:07.984391 25665 tablet_server.cc:178] TabletServer@127.25.16.68:0 shutting down...
I20240502 21:26:07.993136 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:07.993391 25665 tablet_replica.cc:330] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611: stopping tablet replica
I20240502 21:26:07.993566 25665 raft_consensus.cc:2229] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:07.993815 25665 raft_consensus.cc:2258] T bdbac986d7b441818fc3b3038b76bb3e P 07556ecb8098420883d5b64be2f18611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:08.007704 25665 tablet_server.cc:194] TabletServer@127.25.16.68:0 shutdown complete.
I20240502 21:26:08.010568 25665 master.cc:537] Master@127.25.16.126:33931 shutting down...
I20240502 21:26:08.018172 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:08.018383 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:08.018481 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3e00760d1b724a11966281c3a05593ab: stopping tablet replica
I20240502 21:26:08.031680 25665 master.cc:552] Master@127.25.16.126:33931 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26972 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240502 21:26:08.043048 25665 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.16.126:45909
I20240502 21:26:08.043565 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:08.046396 27720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:08.047395 27721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:08.048324 27723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:08.049088 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:08.049330 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:08.049381 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:08.049410 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685168049407 us; error 0 us; skew 500 ppm
I20240502 21:26:08.049552 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:08.055298 25665 webserver.cc:460] Webserver started at http://127.25.16.126:40939/ using document root <none> and password file <none>
I20240502 21:26:08.055558 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:08.055639 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:08.055763 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:08.056463 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714685126025081-25665-0/minicluster-data/master-0-root/instance:
uuid: "2673cf8f1a2b4615b1f0e0db9f6aafd3"
format_stamp: "Formatted at 2024-05-02 21:26:08 on dist-test-slave-b524"
I20240502 21:26:08.059010 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:08.060540 27728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:08.060935 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 21:26:08.061021 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714685126025081-25665-0/minicluster-data/master-0-root
uuid: "2673cf8f1a2b4615b1f0e0db9f6aafd3"
format_stamp: "Formatted at 2024-05-02 21:26:08 on dist-test-slave-b524"
I20240502 21:26:08.061120 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714685126025081-25665-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714685126025081-25665-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714685126025081-25665-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:08.074635 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:08.075188 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:08.086431 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.126:45909
I20240502 21:26:08.086447 27779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.126:45909 every 8 connection(s)
I20240502 21:26:08.087854 27780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:08.093483 27780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3: Bootstrap starting.
I20240502 21:26:08.094830 27780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:08.096423 27780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3: No bootstrap required, opened a new log
I20240502 21:26:08.097220 27780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2673cf8f1a2b4615b1f0e0db9f6aafd3" member_type: VOTER }
I20240502 21:26:08.097348 27780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:08.097388 27780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2673cf8f1a2b4615b1f0e0db9f6aafd3, State: Initialized, Role: FOLLOWER
I20240502 21:26:08.097555 27780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2673cf8f1a2b4615b1f0e0db9f6aafd3" member_type: VOTER }
I20240502 21:26:08.097678 27780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:26:08.097718 27780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:26:08.097765 27780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:08.099373 27780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2673cf8f1a2b4615b1f0e0db9f6aafd3" member_type: VOTER }
I20240502 21:26:08.099543 27780 leader_election.cc:302] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2673cf8f1a2b4615b1f0e0db9f6aafd3; no voters:
I20240502 21:26:08.099859 27780 leader_election.cc:288] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:26:08.099982 27783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:08.100443 27783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [term 1 LEADER]: Becoming Leader. State: Replica: 2673cf8f1a2b4615b1f0e0db9f6aafd3, State: Running, Role: LEADER
I20240502 21:26:08.100687 27780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:26:08.100665 27783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2673cf8f1a2b4615b1f0e0db9f6aafd3" member_type: VOTER }
I20240502 21:26:08.101357 27785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2673cf8f1a2b4615b1f0e0db9f6aafd3. Latest consensus state: current_term: 1 leader_uuid: "2673cf8f1a2b4615b1f0e0db9f6aafd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2673cf8f1a2b4615b1f0e0db9f6aafd3" member_type: VOTER } }
I20240502 21:26:08.101508 27785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:08.101545 27784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2673cf8f1a2b4615b1f0e0db9f6aafd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2673cf8f1a2b4615b1f0e0db9f6aafd3" member_type: VOTER } }
I20240502 21:26:08.101699 27784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:08.104590 27788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:26:08.105751 27788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:26:08.108568 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:26:08.108916 27788 catalog_manager.cc:1347] Generated new cluster ID: 0bae3dbd532148bf88160226a4a733a6
I20240502 21:26:08.108973 27788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:26:08.130266 27788 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:26:08.131048 27788 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:26:08.144222 27788 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3: Generated new TSK 0
I20240502 21:26:08.151082 27788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:26:08.172844 25665 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 8.9407e-05s
I20240502 21:26:20.253573 25665 master.cc:537] Master@127.25.16.126:45909 shutting down...
I20240502 21:26:20.266588 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:20.266834 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:20.266939 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2673cf8f1a2b4615b1f0e0db9f6aafd3: stopping tablet replica
I20240502 21:26:20.281152 25665 master.cc:552] Master@127.25.16.126:45909 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12246 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240502 21:26:20.289057 25665 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.16.126:44283
I20240502 21:26:20.289538 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:20.292129 27802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.292851 27803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.293057 27805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:20.295645 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:20.297449 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:20.297525 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:20.297587 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685180297569 us; error 0 us; skew 500 ppm
I20240502 21:26:20.297746 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:20.305192 25665 webserver.cc:460] Webserver started at http://127.25.16.126:40403/ using document root <none> and password file <none>
I20240502 21:26:20.305411 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:20.305487 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:20.305629 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:20.306214 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/master-0-root/instance:
uuid: "1e75e05bf4dc47e2aefe15a5736379de"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-b524"
I20240502 21:26:20.308516 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:26:20.309928 27810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.310284 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:20.310379 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/master-0-root
uuid: "1e75e05bf4dc47e2aefe15a5736379de"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-b524"
I20240502 21:26:20.310484 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:20.326535 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:20.326947 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:20.337348 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.126:44283
I20240502 21:26:20.337373 27862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.126:44283 every 8 connection(s)
I20240502 21:26:20.338526 27863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.342447 27863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de: Bootstrap starting.
I20240502 21:26:20.343449 27863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:20.344766 27863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de: No bootstrap required, opened a new log
I20240502 21:26:20.345324 27863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e75e05bf4dc47e2aefe15a5736379de" member_type: VOTER }
I20240502 21:26:20.345449 27863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:20.345502 27863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e75e05bf4dc47e2aefe15a5736379de, State: Initialized, Role: FOLLOWER
I20240502 21:26:20.345682 27863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e75e05bf4dc47e2aefe15a5736379de" member_type: VOTER }
I20240502 21:26:20.345796 27863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:26:20.345849 27863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:26:20.345911 27863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:20.347545 27863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e75e05bf4dc47e2aefe15a5736379de" member_type: VOTER }
I20240502 21:26:20.347725 27863 leader_election.cc:302] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e75e05bf4dc47e2aefe15a5736379de; no voters:
I20240502 21:26:20.348044 27863 leader_election.cc:288] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:26:20.348145 27866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:20.348591 27866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [term 1 LEADER]: Becoming Leader. State: Replica: 1e75e05bf4dc47e2aefe15a5736379de, State: Running, Role: LEADER
I20240502 21:26:20.348834 27863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:26:20.348814 27866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e75e05bf4dc47e2aefe15a5736379de" member_type: VOTER }
I20240502 21:26:20.349517 27868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e75e05bf4dc47e2aefe15a5736379de. Latest consensus state: current_term: 1 leader_uuid: "1e75e05bf4dc47e2aefe15a5736379de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e75e05bf4dc47e2aefe15a5736379de" member_type: VOTER } }
I20240502 21:26:20.349566 27867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e75e05bf4dc47e2aefe15a5736379de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e75e05bf4dc47e2aefe15a5736379de" member_type: VOTER } }
I20240502 21:26:20.349792 27868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:20.349844 27867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:20.351069 27871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:26:20.352490 27871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:26:20.353943 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:26:20.355278 27871 catalog_manager.cc:1347] Generated new cluster ID: 5a3d45daba5d428ca4271b11f34cb72d
I20240502 21:26:20.355391 27871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:26:20.364315 27871 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:26:20.365065 27871 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:26:20.381485 27871 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de: Generated new TSK 0
I20240502 21:26:20.381705 27871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:26:20.386435 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:20.389253 27884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.389703 27885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:20.390178 25665 server_base.cc:1034] running on GCE node
W20240502 21:26:20.390408 27887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:20.390692 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:20.390758 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:20.390798 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685180390784 us; error 0 us; skew 500 ppm
I20240502 21:26:20.390915 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:20.392021 25665 webserver.cc:460] Webserver started at http://127.25.16.65:39779/ using document root <none> and password file <none>
I20240502 21:26:20.392215 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:20.392309 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:20.392438 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:20.393013 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-0-root/instance:
uuid: "0b52f76b6cb04eed862ef594caaf2379"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-b524"
I20240502 21:26:20.395242 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 21:26:20.396667 27892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.396989 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 21:26:20.397095 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-0-root
uuid: "0b52f76b6cb04eed862ef594caaf2379"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-b524"
I20240502 21:26:20.397210 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:20.405366 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:20.405798 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:20.406271 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:20.407075 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:20.407146 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.407236 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:20.407290 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.419796 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.65:42175
I20240502 21:26:20.419818 27954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.65:42175 every 8 connection(s)
I20240502 21:26:20.421216 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:20.424098 27959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.424571 27960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.426808 27962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:20.427313 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:20.427561 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:20.427641 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:20.427676 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685180427662 us; error 0 us; skew 500 ppm
I20240502 21:26:20.427832 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:20.429201 27955 heartbeater.cc:347] Connected to a master server at 127.25.16.126:44283
I20240502 21:26:20.429325 27955 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:20.429633 27955 heartbeater.cc:511] Master 127.25.16.126:44283 requested a full tablet report, sending...
I20240502 21:26:20.430608 27827 ts_manager.cc:194] Registered new tserver with Master: 0b52f76b6cb04eed862ef594caaf2379 (127.25.16.65:42175)
I20240502 21:26:20.432174 27827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43830
I20240502 21:26:20.435214 25665 webserver.cc:460] Webserver started at http://127.25.16.66:43931/ using document root <none> and password file <none>
I20240502 21:26:20.435461 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:20.435531 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:20.435662 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:20.436203 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-1-root/instance:
uuid: "1d8d5a1c5e3049e288e9cdbed230450b"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-b524"
I20240502 21:26:20.438409 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:20.439817 27967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.440156 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:20.440259 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-1-root
uuid: "1d8d5a1c5e3049e288e9cdbed230450b"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-b524"
I20240502 21:26:20.440393 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:20.449174 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:20.449553 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:20.450032 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:20.450748 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:20.450815 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.450901 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:20.450956 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.463142 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.66:36365
I20240502 21:26:20.463167 28030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.66:36365 every 8 connection(s)
I20240502 21:26:20.464550 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:20.467383 28034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.467911 28035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:20.468761 28037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:20.470479 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:20.470973 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:20.471048 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:20.471110 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685180471087 us; error 0 us; skew 500 ppm
I20240502 21:26:20.471275 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:20.472472 28031 heartbeater.cc:347] Connected to a master server at 127.25.16.126:44283
I20240502 21:26:20.472625 28031 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:20.472698 25665 webserver.cc:460] Webserver started at http://127.25.16.67:38733/ using document root <none> and password file <none>
I20240502 21:26:20.472929 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:20.472973 28031 heartbeater.cc:511] Master 127.25.16.126:44283 requested a full tablet report, sending...
I20240502 21:26:20.473016 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:20.473379 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:20.473832 27827 ts_manager.cc:194] Registered new tserver with Master: 1d8d5a1c5e3049e288e9cdbed230450b (127.25.16.66:36365)
I20240502 21:26:20.474184 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-2-root/instance:
uuid: "76eec3e34d734445beb1e8f0da74fde9"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-b524"
I20240502 21:26:20.474589 27827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43836
I20240502 21:26:20.477039 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:20.478454 28042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.478839 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:20.478957 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-2-root
uuid: "76eec3e34d734445beb1e8f0da74fde9"
format_stamp: "Formatted at 2024-05-02 21:26:20 on dist-test-slave-b524"
I20240502 21:26:20.479069 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714685126025081-25665-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:20.492893 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:20.493461 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:20.493942 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:20.494654 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:20.494720 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.494809 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:20.494858 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:20.507030 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.67:34027
I20240502 21:26:20.507063 28104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.67:34027 every 8 connection(s)
I20240502 21:26:20.513227 28105 heartbeater.cc:347] Connected to a master server at 127.25.16.126:44283
I20240502 21:26:20.513351 28105 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:20.513654 28105 heartbeater.cc:511] Master 127.25.16.126:44283 requested a full tablet report, sending...
I20240502 21:26:20.514425 27827 ts_manager.cc:194] Registered new tserver with Master: 76eec3e34d734445beb1e8f0da74fde9 (127.25.16.67:34027)
I20240502 21:26:20.514611 25665 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006772387s
I20240502 21:26:20.515241 27827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43844
I20240502 21:26:21.433434 27955 heartbeater.cc:503] Master 127.25.16.126:44283 was elected leader, sending a full tablet report...
I20240502 21:26:21.475960 28031 heartbeater.cc:503] Master 127.25.16.126:44283 was elected leader, sending a full tablet report...
I20240502 21:26:21.516433 28105 heartbeater.cc:503] Master 127.25.16.126:44283 was elected leader, sending a full tablet report...
I20240502 21:26:22.353863 27881 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:24.354266 27881 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:26.354733 27881 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:28.355171 27881 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:30.355598 27881 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:32.356042 27881 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:32.594380 25665 tablet_server.cc:178] TabletServer@127.25.16.65:0 shutting down...
I20240502 21:26:32.605365 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:32.618326 25665 tablet_server.cc:194] TabletServer@127.25.16.65:0 shutdown complete.
I20240502 21:26:32.620913 25665 tablet_server.cc:178] TabletServer@127.25.16.66:0 shutting down...
I20240502 21:26:32.628351 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:32.640816 25665 tablet_server.cc:194] TabletServer@127.25.16.66:0 shutdown complete.
I20240502 21:26:32.643304 25665 tablet_server.cc:178] TabletServer@127.25.16.67:0 shutting down...
I20240502 21:26:32.663327 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:32.675580 25665 tablet_server.cc:194] TabletServer@127.25.16.67:0 shutdown complete.
I20240502 21:26:32.678237 25665 master.cc:537] Master@127.25.16.126:44283 shutting down...
I20240502 21:26:32.684983 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:32.685186 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:32.685281 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1e75e05bf4dc47e2aefe15a5736379de: stopping tablet replica
I20240502 21:26:32.697974 25665 master.cc:552] Master@127.25.16.126:44283 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12417 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240502 21:26:32.706605 25665 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.16.126:39387
I20240502 21:26:32.707052 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:32.709424 28109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:32.709720 28110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:32.710224 28112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:32.711098 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:32.711362 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:32.711414 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:32.711455 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685192711451 us; error 0 us; skew 500 ppm
I20240502 21:26:32.711596 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:32.712848 25665 webserver.cc:460] Webserver started at http://127.25.16.126:38057/ using document root <none> and password file <none>
I20240502 21:26:32.713073 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:32.713127 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:32.713239 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:32.713841 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/master-0-root/instance:
uuid: "e2d26dd1686846bb80d12e29c7c0438b"
format_stamp: "Formatted at 2024-05-02 21:26:32 on dist-test-slave-b524"
I20240502 21:26:32.716167 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:32.717646 28117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:32.717936 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:32.718020 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/master-0-root
uuid: "e2d26dd1686846bb80d12e29c7c0438b"
format_stamp: "Formatted at 2024-05-02 21:26:32 on dist-test-slave-b524"
I20240502 21:26:32.718117 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:32.741017 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:32.741472 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:32.751237 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.126:39387
I20240502 21:26:32.751255 28168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.126:39387 every 8 connection(s)
I20240502 21:26:32.752410 28169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.756389 28169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b: Bootstrap starting.
I20240502 21:26:32.757400 28169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:32.758682 28169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b: No bootstrap required, opened a new log
I20240502 21:26:32.759193 28169 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d26dd1686846bb80d12e29c7c0438b" member_type: VOTER }
I20240502 21:26:32.759307 28169 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:32.759356 28169 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d26dd1686846bb80d12e29c7c0438b, State: Initialized, Role: FOLLOWER
I20240502 21:26:32.759526 28169 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d26dd1686846bb80d12e29c7c0438b" member_type: VOTER }
I20240502 21:26:32.759635 28169 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:26:32.759689 28169 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:26:32.759754 28169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.761430 28169 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d26dd1686846bb80d12e29c7c0438b" member_type: VOTER }
I20240502 21:26:32.761600 28169 leader_election.cc:302] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2d26dd1686846bb80d12e29c7c0438b; no voters:
I20240502 21:26:32.761935 28169 leader_election.cc:288] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:26:32.762020 28172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:32.762394 28172 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [term 1 LEADER]: Becoming Leader. State: Replica: e2d26dd1686846bb80d12e29c7c0438b, State: Running, Role: LEADER
I20240502 21:26:32.762624 28172 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d26dd1686846bb80d12e29c7c0438b" member_type: VOTER }
I20240502 21:26:32.762701 28169 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:26:32.763310 28174 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2d26dd1686846bb80d12e29c7c0438b. Latest consensus state: current_term: 1 leader_uuid: "e2d26dd1686846bb80d12e29c7c0438b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d26dd1686846bb80d12e29c7c0438b" member_type: VOTER } }
I20240502 21:26:32.763259 28173 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2d26dd1686846bb80d12e29c7c0438b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d26dd1686846bb80d12e29c7c0438b" member_type: VOTER } }
I20240502 21:26:32.763439 28174 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:32.763463 28173 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:32.764358 28176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:26:32.765690 28176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:26:32.767213 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:26:32.768432 28176 catalog_manager.cc:1347] Generated new cluster ID: 4048dc7926e0434c8ecb85014eca327c
I20240502 21:26:32.768491 28176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:26:32.809576 28176 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:26:32.810294 28176 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:26:32.822160 28176 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b: Generated new TSK 0
I20240502 21:26:32.822356 28176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:26:32.831883 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:32.834579 28190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:32.835031 28191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:32.835399 28193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:32.836328 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:32.836563 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:32.836614 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:32.836644 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685192836640 us; error 0 us; skew 500 ppm
I20240502 21:26:32.836781 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:32.837929 25665 webserver.cc:460] Webserver started at http://127.25.16.65:34795/ using document root <none> and password file <none>
I20240502 21:26:32.838140 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:32.838201 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:32.838320 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:32.838908 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-0-root/instance:
uuid: "a860b105511a4a57a65d7fb3a19d13a1"
format_stamp: "Formatted at 2024-05-02 21:26:32 on dist-test-slave-b524"
I20240502 21:26:32.841260 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:32.842701 28198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:32.843000 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:32.843088 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-0-root
uuid: "a860b105511a4a57a65d7fb3a19d13a1"
format_stamp: "Formatted at 2024-05-02 21:26:32 on dist-test-slave-b524"
I20240502 21:26:32.843185 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:32.851073 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:32.851485 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:32.851913 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:32.852715 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:32.852767 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:32.852833 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:32.852872 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:32.863235 28260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.65:43825 every 8 connection(s)
I20240502 21:26:32.863240 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.65:43825
I20240502 21:26:32.864746 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:32.867492 28266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:32.867944 28267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:32.869771 28269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:32.870400 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:32.870688 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:32.870751 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:32.870785 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685192870782 us; error 0 us; skew 500 ppm
I20240502 21:26:32.870918 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:32.872119 25665 webserver.cc:460] Webserver started at http://127.25.16.66:33547/ using document root <none> and password file <none>
I20240502 21:26:32.872362 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:32.872437 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:32.872545 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:32.873234 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-1-root/instance:
uuid: "df567eadea6a439c89e862cec821f0ac"
format_stamp: "Formatted at 2024-05-02 21:26:32 on dist-test-slave-b524"
I20240502 21:26:32.874239 28261 heartbeater.cc:347] Connected to a master server at 127.25.16.126:39387
I20240502 21:26:32.874359 28261 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:32.874704 28261 heartbeater.cc:511] Master 127.25.16.126:39387 requested a full tablet report, sending...
I20240502 21:26:32.875566 28134 ts_manager.cc:194] Registered new tserver with Master: a860b105511a4a57a65d7fb3a19d13a1 (127.25.16.65:43825)
I20240502 21:26:32.875965 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:26:32.876981 28134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32786
I20240502 21:26:32.877784 28274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:32.878103 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:32.878181 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-1-root
uuid: "df567eadea6a439c89e862cec821f0ac"
format_stamp: "Formatted at 2024-05-02 21:26:32 on dist-test-slave-b524"
I20240502 21:26:32.878273 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:32.901048 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:32.901510 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:32.901957 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:32.902678 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:32.902730 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:32.902789 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:32.902822 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:32.913697 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.66:36393
I20240502 21:26:32.913724 28337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.66:36393 every 8 connection(s)
I20240502 21:26:32.915102 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:32.917737 28341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:32.918192 28342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:32.918694 28344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:32.921304 28338 heartbeater.cc:347] Connected to a master server at 127.25.16.126:39387
I20240502 21:26:32.921414 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:32.921423 28338 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:32.921722 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:32.921792 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:32.921833 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685192921829 us; error 0 us; skew 500 ppm
I20240502 21:26:32.921860 28338 heartbeater.cc:511] Master 127.25.16.126:39387 requested a full tablet report, sending...
I20240502 21:26:32.921978 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:32.922705 28134 ts_manager.cc:194] Registered new tserver with Master: df567eadea6a439c89e862cec821f0ac (127.25.16.66:36393)
I20240502 21:26:32.923194 25665 webserver.cc:460] Webserver started at http://127.25.16.67:45129/ using document root <none> and password file <none>
I20240502 21:26:32.923444 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:32.923508 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:32.923609 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:32.923658 28134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32794
I20240502 21:26:32.924315 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-2-root/instance:
uuid: "8bab345d40134a299b2b6eb204ea3b49"
format_stamp: "Formatted at 2024-05-02 21:26:32 on dist-test-slave-b524"
I20240502 21:26:32.926517 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:32.927865 28349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:32.928150 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:32.928231 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-2-root
uuid: "8bab345d40134a299b2b6eb204ea3b49"
format_stamp: "Formatted at 2024-05-02 21:26:32 on dist-test-slave-b524"
I20240502 21:26:32.928349 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:32.942157 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:32.942626 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:32.943070 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:32.943768 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:32.943818 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:32.943866 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:32.943905 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:32.955225 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.67:39387
I20240502 21:26:32.955258 28411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.67:39387 every 8 connection(s)
I20240502 21:26:32.956583 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:32.959105 28415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:32.959553 28416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:32.959971 28418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:32.962607 28412 heartbeater.cc:347] Connected to a master server at 127.25.16.126:39387
I20240502 21:26:32.962723 28412 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:32.963016 28412 heartbeater.cc:511] Master 127.25.16.126:39387 requested a full tablet report, sending...
I20240502 21:26:32.963771 28134 ts_manager.cc:194] Registered new tserver with Master: 8bab345d40134a299b2b6eb204ea3b49 (127.25.16.67:39387)
I20240502 21:26:32.964466 28134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32806
I20240502 21:26:32.966495 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:32.966743 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:32.966794 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:32.966830 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685192966825 us; error 0 us; skew 500 ppm
I20240502 21:26:32.966964 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:32.968077 25665 webserver.cc:460] Webserver started at http://127.25.16.68:35585/ using document root <none> and password file <none>
I20240502 21:26:32.968268 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:32.968374 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:32.968498 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:32.969071 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-3-root/instance:
uuid: "1c4bc32702b84985872caf55f7393870"
format_stamp: "Formatted at 2024-05-02 21:26:32 on dist-test-slave-b524"
I20240502 21:26:32.971299 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 21:26:32.972752 28423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:32.973107 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 21:26:32.973204 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-3-root
uuid: "1c4bc32702b84985872caf55f7393870"
format_stamp: "Formatted at 2024-05-02 21:26:32 on dist-test-slave-b524"
I20240502 21:26:32.973309 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714685126025081-25665-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:32.990195 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:32.990674 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:32.991142 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:32.991869 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:32.991943 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:32.992012 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:32.992053 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:33.003404 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.68:43167
I20240502 21:26:33.003443 28485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.68:43167 every 8 connection(s)
I20240502 21:26:33.009454 28486 heartbeater.cc:347] Connected to a master server at 127.25.16.126:39387
I20240502 21:26:33.009589 28486 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:33.009936 28486 heartbeater.cc:511] Master 127.25.16.126:39387 requested a full tablet report, sending...
I20240502 21:26:33.010725 28134 ts_manager.cc:194] Registered new tserver with Master: 1c4bc32702b84985872caf55f7393870 (127.25.16.68:43167)
I20240502 21:26:33.011051 25665 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.006839262s
I20240502 21:26:33.011763 28134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32808
I20240502 21:26:33.878265 28261 heartbeater.cc:503] Master 127.25.16.126:39387 was elected leader, sending a full tablet report...
I20240502 21:26:33.924803 28338 heartbeater.cc:503] Master 127.25.16.126:39387 was elected leader, sending a full tablet report...
I20240502 21:26:33.965593 28412 heartbeater.cc:503] Master 127.25.16.126:39387 was elected leader, sending a full tablet report...
I20240502 21:26:34.013015 28486 heartbeater.cc:503] Master 127.25.16.126:39387 was elected leader, sending a full tablet report...
I20240502 21:26:34.044965 25665 test_util.cc:268] Using random seed: 1105495581
I20240502 21:26:34.058228 28134 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32822:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 21:26:34.073391 28226 tablet_service.cc:1467] Processing CreateTablet for tablet bcb2b30c58904d61854314ab6e1d2484 (DEFAULT_TABLE table=test-workload [id=0fe09b9f85ba4e34bfde22093375c78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.073889 28226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb2b30c58904d61854314ab6e1d2484. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.075546 28451 tablet_service.cc:1467] Processing CreateTablet for tablet bcb2b30c58904d61854314ab6e1d2484 (DEFAULT_TABLE table=test-workload [id=0fe09b9f85ba4e34bfde22093375c78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.076026 28451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb2b30c58904d61854314ab6e1d2484. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.078011 28377 tablet_service.cc:1467] Processing CreateTablet for tablet bcb2b30c58904d61854314ab6e1d2484 (DEFAULT_TABLE table=test-workload [id=0fe09b9f85ba4e34bfde22093375c78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.078420 28377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb2b30c58904d61854314ab6e1d2484. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.081467 28508 tablet_bootstrap.cc:492] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870: Bootstrap starting.
I20240502 21:26:34.082175 28507 tablet_bootstrap.cc:492] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1: Bootstrap starting.
I20240502 21:26:34.083397 28508 tablet_bootstrap.cc:654] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.083904 28507 tablet_bootstrap.cc:654] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.084182 28509 tablet_bootstrap.cc:492] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49: Bootstrap starting.
I20240502 21:26:34.085873 28509 tablet_bootstrap.cc:654] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.086148 28507 tablet_bootstrap.cc:492] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1: No bootstrap required, opened a new log
I20240502 21:26:34.086149 28508 tablet_bootstrap.cc:492] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870: No bootstrap required, opened a new log
I20240502 21:26:34.086340 28508 ts_tablet_manager.cc:1367] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:26:34.086337 28507 ts_tablet_manager.cc:1367] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:34.087165 28507 raft_consensus.cc:348] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c4bc32702b84985872caf55f7393870" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 43167 } } peers { permanent_uuid: "a860b105511a4a57a65d7fb3a19d13a1" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 43825 } } peers { permanent_uuid: "8bab345d40134a299b2b6eb204ea3b49" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 39387 } }
I20240502 21:26:34.087167 28508 raft_consensus.cc:348] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c4bc32702b84985872caf55f7393870" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 43167 } } peers { permanent_uuid: "a860b105511a4a57a65d7fb3a19d13a1" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 43825 } } peers { permanent_uuid: "8bab345d40134a299b2b6eb204ea3b49" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 39387 } }
I20240502 21:26:34.087425 28508 raft_consensus.cc:374] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.087520 28509 tablet_bootstrap.cc:492] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49: No bootstrap required, opened a new log
I20240502 21:26:34.087422 28507 raft_consensus.cc:374] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.087570 28508 raft_consensus.cc:729] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c4bc32702b84985872caf55f7393870, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.087703 28507 raft_consensus.cc:729] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a860b105511a4a57a65d7fb3a19d13a1, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.087706 28509 ts_tablet_manager.cc:1367] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:34.087953 28507 consensus_queue.cc:260] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c4bc32702b84985872caf55f7393870" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 43167 } } peers { permanent_uuid: "a860b105511a4a57a65d7fb3a19d13a1" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 43825 } } peers { permanent_uuid: "8bab345d40134a299b2b6eb204ea3b49" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 39387 } }
I20240502 21:26:34.088464 28509 raft_consensus.cc:348] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c4bc32702b84985872caf55f7393870" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 43167 } } peers { permanent_uuid: "a860b105511a4a57a65d7fb3a19d13a1" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 43825 } } peers { permanent_uuid: "8bab345d40134a299b2b6eb204ea3b49" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 39387 } }
I20240502 21:26:34.088639 28509 raft_consensus.cc:374] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.088766 28509 raft_consensus.cc:729] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bab345d40134a299b2b6eb204ea3b49, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.088773 28507 ts_tablet_manager.cc:1398] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:34.087945 28508 consensus_queue.cc:260] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c4bc32702b84985872caf55f7393870" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 43167 } } peers { permanent_uuid: "a860b105511a4a57a65d7fb3a19d13a1" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 43825 } } peers { permanent_uuid: "8bab345d40134a299b2b6eb204ea3b49" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 39387 } }
I20240502 21:26:34.089149 28509 consensus_queue.cc:260] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c4bc32702b84985872caf55f7393870" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 43167 } } peers { permanent_uuid: "a860b105511a4a57a65d7fb3a19d13a1" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 43825 } } peers { permanent_uuid: "8bab345d40134a299b2b6eb204ea3b49" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 39387 } }
I20240502 21:26:34.090368 28508 ts_tablet_manager.cc:1398] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:34.090919 28509 ts_tablet_manager.cc:1398] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20240502 21:26:34.114986 28262 tablet.cc:2369] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:34.205384 28513 raft_consensus.cc:482] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:34.205574 28513 raft_consensus.cc:504] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c4bc32702b84985872caf55f7393870" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 43167 } } peers { permanent_uuid: "a860b105511a4a57a65d7fb3a19d13a1" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 43825 } } peers { permanent_uuid: "8bab345d40134a299b2b6eb204ea3b49" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 39387 } }
I20240502 21:26:34.206354 28513 leader_election.cc:288] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c4bc32702b84985872caf55f7393870 (127.25.16.68:43167), 8bab345d40134a299b2b6eb204ea3b49 (127.25.16.67:39387)
I20240502 21:26:34.212179 28461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb2b30c58904d61854314ab6e1d2484" candidate_uuid: "a860b105511a4a57a65d7fb3a19d13a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c4bc32702b84985872caf55f7393870" is_pre_election: true
I20240502 21:26:34.212227 28387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb2b30c58904d61854314ab6e1d2484" candidate_uuid: "a860b105511a4a57a65d7fb3a19d13a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bab345d40134a299b2b6eb204ea3b49" is_pre_election: true
I20240502 21:26:34.212405 28461 raft_consensus.cc:2454] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a860b105511a4a57a65d7fb3a19d13a1 in term 0.
I20240502 21:26:34.212481 28387 raft_consensus.cc:2454] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a860b105511a4a57a65d7fb3a19d13a1 in term 0.
I20240502 21:26:34.212810 28200 leader_election.cc:302] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c4bc32702b84985872caf55f7393870, a860b105511a4a57a65d7fb3a19d13a1; no voters:
I20240502 21:26:34.213073 28513 raft_consensus.cc:2781] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:34.213155 28513 raft_consensus.cc:482] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:34.213233 28513 raft_consensus.cc:3037] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.215000 28513 raft_consensus.cc:504] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c4bc32702b84985872caf55f7393870" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 43167 } } peers { permanent_uuid: "a860b105511a4a57a65d7fb3a19d13a1" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 43825 } } peers { permanent_uuid: "8bab345d40134a299b2b6eb204ea3b49" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 39387 } }
I20240502 21:26:34.215481 28513 leader_election.cc:288] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [CANDIDATE]: Term 1 election: Requested vote from peers 1c4bc32702b84985872caf55f7393870 (127.25.16.68:43167), 8bab345d40134a299b2b6eb204ea3b49 (127.25.16.67:39387)
I20240502 21:26:34.215835 28387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb2b30c58904d61854314ab6e1d2484" candidate_uuid: "a860b105511a4a57a65d7fb3a19d13a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bab345d40134a299b2b6eb204ea3b49"
I20240502 21:26:34.215837 28461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb2b30c58904d61854314ab6e1d2484" candidate_uuid: "a860b105511a4a57a65d7fb3a19d13a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c4bc32702b84985872caf55f7393870"
I20240502 21:26:34.216063 28387 raft_consensus.cc:3037] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.216147 28461 raft_consensus.cc:3037] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.217900 28387 raft_consensus.cc:2454] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a860b105511a4a57a65d7fb3a19d13a1 in term 1.
I20240502 21:26:34.217991 28461 raft_consensus.cc:2454] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a860b105511a4a57a65d7fb3a19d13a1 in term 1.
I20240502 21:26:34.218261 28199 leader_election.cc:302] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bab345d40134a299b2b6eb204ea3b49, a860b105511a4a57a65d7fb3a19d13a1; no voters:
I20240502 21:26:34.218559 28513 raft_consensus.cc:2781] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:34.218933 28513 raft_consensus.cc:686] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 1 LEADER]: Becoming Leader. State: Replica: a860b105511a4a57a65d7fb3a19d13a1, State: Running, Role: LEADER
I20240502 21:26:34.219197 28513 consensus_queue.cc:237] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c4bc32702b84985872caf55f7393870" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 43167 } } peers { permanent_uuid: "a860b105511a4a57a65d7fb3a19d13a1" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 43825 } } peers { permanent_uuid: "8bab345d40134a299b2b6eb204ea3b49" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 39387 } }
I20240502 21:26:34.221536 28132 catalog_manager.cc:5484] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a860b105511a4a57a65d7fb3a19d13a1 (127.25.16.65). New cstate: current_term: 1 leader_uuid: "a860b105511a4a57a65d7fb3a19d13a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c4bc32702b84985872caf55f7393870" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 43167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a860b105511a4a57a65d7fb3a19d13a1" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 43825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bab345d40134a299b2b6eb204ea3b49" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 39387 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:34.767254 28187 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:34.767784 28187 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:34.767935 28187 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:34.775565 28520 consensus_queue.cc:1035] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c4bc32702b84985872caf55f7393870" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 43167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:34.781080 28523 consensus_queue.cc:1035] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bab345d40134a299b2b6eb204ea3b49" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 39387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:36.768365 28187 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:36.768846 28187 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:36.768966 28187 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:38.769333 28187 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:38.769768 28187 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:38.769883 28187 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:40.770253 28187 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:40.770712 28187 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:40.770839 28187 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:42.771183 28187 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:42.771579 28187 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:42.771688 28187 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:44.772043 28187 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:44.772480 28187 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:44.772570 28187 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:45.314872 25665 tablet_server.cc:178] TabletServer@127.25.16.65:0 shutting down...
I20240502 21:26:45.331130 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:45.331415 25665 tablet_replica.cc:330] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1: stopping tablet replica
I20240502 21:26:45.331604 25665 raft_consensus.cc:2229] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:45.331967 25665 raft_consensus.cc:2258] T bcb2b30c58904d61854314ab6e1d2484 P a860b105511a4a57a65d7fb3a19d13a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.347270 25665 tablet_server.cc:194] TabletServer@127.25.16.65:0 shutdown complete.
I20240502 21:26:45.350221 25665 tablet_server.cc:178] TabletServer@127.25.16.66:0 shutting down...
I20240502 21:26:45.358160 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:45.370463 25665 tablet_server.cc:194] TabletServer@127.25.16.66:0 shutdown complete.
I20240502 21:26:45.373148 25665 tablet_server.cc:178] TabletServer@127.25.16.67:0 shutting down...
I20240502 21:26:45.381187 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:45.381422 25665 tablet_replica.cc:330] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49: stopping tablet replica
I20240502 21:26:45.381604 25665 raft_consensus.cc:2229] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.381801 25665 raft_consensus.cc:2258] T bcb2b30c58904d61854314ab6e1d2484 P 8bab345d40134a299b2b6eb204ea3b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.394882 25665 tablet_server.cc:194] TabletServer@127.25.16.67:0 shutdown complete.
I20240502 21:26:45.397675 25665 tablet_server.cc:178] TabletServer@127.25.16.68:0 shutting down...
I20240502 21:26:45.406004 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:45.406200 25665 tablet_replica.cc:330] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870: stopping tablet replica
I20240502 21:26:45.406368 25665 raft_consensus.cc:2229] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.406535 25665 raft_consensus.cc:2258] T bcb2b30c58904d61854314ab6e1d2484 P 1c4bc32702b84985872caf55f7393870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.419667 25665 tablet_server.cc:194] TabletServer@127.25.16.68:0 shutdown complete.
I20240502 21:26:45.422462 25665 master.cc:537] Master@127.25.16.126:39387 shutting down...
I20240502 21:26:45.429194 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:45.429405 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.429515 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P e2d26dd1686846bb80d12e29c7c0438b: stopping tablet replica
I20240502 21:26:45.442322 25665 master.cc:552] Master@127.25.16.126:39387 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12747 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240502 21:26:45.454236 25665 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.16.126:40261
I20240502 21:26:45.454705 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:45.457226 28530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:45.457650 28531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:45.458031 28533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:45.458992 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:45.459184 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:45.459244 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:45.459300 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685205459281 us; error 0 us; skew 500 ppm
I20240502 21:26:45.459450 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:45.460638 25665 webserver.cc:460] Webserver started at http://127.25.16.126:42363/ using document root <none> and password file <none>
I20240502 21:26:45.467532 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:45.467602 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:45.467739 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:45.468433 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/master-0-root/instance:
uuid: "16181cda5c49423aa7d39f5f444db8d8"
format_stamp: "Formatted at 2024-05-02 21:26:45 on dist-test-slave-b524"
I20240502 21:26:45.470701 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:45.472081 28538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:45.472429 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:45.472520 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/master-0-root
uuid: "16181cda5c49423aa7d39f5f444db8d8"
format_stamp: "Formatted at 2024-05-02 21:26:45 on dist-test-slave-b524"
I20240502 21:26:45.472608 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:45.484509 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:45.484897 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:45.494777 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.126:40261
I20240502 21:26:45.494803 28589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.126:40261 every 8 connection(s)
I20240502 21:26:45.495887 28590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.499728 28590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8: Bootstrap starting.
I20240502 21:26:45.500777 28590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.502003 28590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8: No bootstrap required, opened a new log
I20240502 21:26:45.502525 28590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16181cda5c49423aa7d39f5f444db8d8" member_type: VOTER }
I20240502 21:26:45.502645 28590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.502701 28590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16181cda5c49423aa7d39f5f444db8d8, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.502871 28590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16181cda5c49423aa7d39f5f444db8d8" member_type: VOTER }
I20240502 21:26:45.502980 28590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:26:45.503036 28590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:26:45.503101 28590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.504799 28590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16181cda5c49423aa7d39f5f444db8d8" member_type: VOTER }
I20240502 21:26:45.504985 28590 leader_election.cc:302] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16181cda5c49423aa7d39f5f444db8d8; no voters:
I20240502 21:26:45.505302 28590 leader_election.cc:288] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:26:45.505393 28593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:45.505765 28593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [term 1 LEADER]: Becoming Leader. State: Replica: 16181cda5c49423aa7d39f5f444db8d8, State: Running, Role: LEADER
I20240502 21:26:45.505964 28593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16181cda5c49423aa7d39f5f444db8d8" member_type: VOTER }
I20240502 21:26:45.506031 28590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:26:45.506570 28594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16181cda5c49423aa7d39f5f444db8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16181cda5c49423aa7d39f5f444db8d8" member_type: VOTER } }
I20240502 21:26:45.506611 28595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16181cda5c49423aa7d39f5f444db8d8. Latest consensus state: current_term: 1 leader_uuid: "16181cda5c49423aa7d39f5f444db8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16181cda5c49423aa7d39f5f444db8d8" member_type: VOTER } }
I20240502 21:26:45.506757 28594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:45.506858 28595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:45.507635 28598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:26:45.508857 28598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:26:45.510725 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:26:45.511523 28598 catalog_manager.cc:1347] Generated new cluster ID: f8ff9a1fdba44db894b65f92ac1eb9f8
I20240502 21:26:45.511598 28598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:26:45.541311 28598 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:26:45.542016 28598 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:26:45.553987 28598 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8: Generated new TSK 0
I20240502 21:26:45.554174 28598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:26:45.575563 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:45.578208 28611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:45.578621 28612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:45.579070 28614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:45.579891 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:45.580082 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:45.580132 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:45.580166 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685205580162 us; error 0 us; skew 500 ppm
I20240502 21:26:45.580334 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:45.581463 25665 webserver.cc:460] Webserver started at http://127.25.16.65:38301/ using document root <none> and password file <none>
I20240502 21:26:45.581676 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:45.581732 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:45.581844 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:45.582408 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-0-root/instance:
uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9"
format_stamp: "Formatted at 2024-05-02 21:26:45 on dist-test-slave-b524"
I20240502 21:26:45.584707 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 21:26:45.586078 28619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:45.586367 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240502 21:26:45.586455 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-0-root
uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9"
format_stamp: "Formatted at 2024-05-02 21:26:45 on dist-test-slave-b524"
I20240502 21:26:45.586555 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:45.610394 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:45.610867 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:45.611339 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:45.612118 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:45.612170 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:45.612236 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:45.612269 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:45.623160 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.65:44255
I20240502 21:26:45.623191 28681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.65:44255 every 8 connection(s)
I20240502 21:26:45.624604 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:45.627135 28686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:45.627529 28687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:45.627926 28689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:45.628454 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:45.628677 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:45.628728 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:45.628758 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685205628754 us; error 0 us; skew 500 ppm
I20240502 21:26:45.628892 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:45.629997 25665 webserver.cc:460] Webserver started at http://127.25.16.66:38531/ using document root <none> and password file <none>
I20240502 21:26:45.630157 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:45.630201 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:45.630277 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:45.630780 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-1-root/instance:
uuid: "e460e5c286b54472994554cc72c536ca"
format_stamp: "Formatted at 2024-05-02 21:26:45 on dist-test-slave-b524"
I20240502 21:26:45.632982 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:45.633571 28682 heartbeater.cc:347] Connected to a master server at 127.25.16.126:40261
I20240502 21:26:45.633680 28682 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:45.634011 28682 heartbeater.cc:511] Master 127.25.16.126:40261 requested a full tablet report, sending...
I20240502 21:26:45.634521 28694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:45.634830 28555 ts_manager.cc:194] Registered new tserver with Master: 1fedd5f8f9434e8691d930d5cb5c3bd9 (127.25.16.65:44255)
I20240502 21:26:45.634835 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:45.634940 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-1-root
uuid: "e460e5c286b54472994554cc72c536ca"
format_stamp: "Formatted at 2024-05-02 21:26:45 on dist-test-slave-b524"
I20240502 21:26:45.635049 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:45.635861 28555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54504
I20240502 21:26:45.645285 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:45.645663 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:45.646090 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:45.646742 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:45.646795 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:45.646852 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:45.646885 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:45.658351 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.66:35097
I20240502 21:26:45.658375 28757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.66:35097 every 8 connection(s)
I20240502 21:26:45.659674 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:45.662180 28761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:45.662603 28762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:45.663050 25665 server_base.cc:1034] running on GCE node
W20240502 21:26:45.663609 28764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:45.663893 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:45.663971 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:45.664017 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685205664013 us; error 0 us; skew 500 ppm
I20240502 21:26:45.664152 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:45.665335 25665 webserver.cc:460] Webserver started at http://127.25.16.67:44357/ using document root <none> and password file <none>
I20240502 21:26:45.665429 28758 heartbeater.cc:347] Connected to a master server at 127.25.16.126:40261
I20240502 21:26:45.665534 28758 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:45.665532 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:45.665617 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:45.665727 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:45.665870 28758 heartbeater.cc:511] Master 127.25.16.126:40261 requested a full tablet report, sending...
I20240502 21:26:45.666335 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-2-root/instance:
uuid: "1facdf918c044d84a9f42c58520282d4"
format_stamp: "Formatted at 2024-05-02 21:26:45 on dist-test-slave-b524"
I20240502 21:26:45.666626 28555 ts_manager.cc:194] Registered new tserver with Master: e460e5c286b54472994554cc72c536ca (127.25.16.66:35097)
I20240502 21:26:45.667282 28555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54512
I20240502 21:26:45.668606 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:45.669889 28769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:45.670217 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:45.670296 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-2-root
uuid: "1facdf918c044d84a9f42c58520282d4"
format_stamp: "Formatted at 2024-05-02 21:26:45 on dist-test-slave-b524"
I20240502 21:26:45.670374 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714685126025081-25665-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:45.687695 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:45.688060 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:45.688513 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:45.689179 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:45.689229 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:45.689275 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:45.689302 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:45.700423 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.67:45755
I20240502 21:26:45.700449 28832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.67:45755 every 8 connection(s)
I20240502 21:26:45.706233 28833 heartbeater.cc:347] Connected to a master server at 127.25.16.126:40261
I20240502 21:26:45.706344 28833 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:45.706660 28833 heartbeater.cc:511] Master 127.25.16.126:40261 requested a full tablet report, sending...
I20240502 21:26:45.707444 28555 ts_manager.cc:194] Registered new tserver with Master: 1facdf918c044d84a9f42c58520282d4 (127.25.16.67:45755)
I20240502 21:26:45.708022 25665 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006759055s
I20240502 21:26:45.708192 28555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54524
I20240502 21:26:46.637205 28682 heartbeater.cc:503] Master 127.25.16.126:40261 was elected leader, sending a full tablet report...
I20240502 21:26:46.668334 28758 heartbeater.cc:503] Master 127.25.16.126:40261 was elected leader, sending a full tablet report...
I20240502 21:26:46.709426 28833 heartbeater.cc:503] Master 127.25.16.126:40261 was elected leader, sending a full tablet report...
I20240502 21:26:46.740990 25665 test_util.cc:268] Using random seed: 1118191608
I20240502 21:26:46.750669 28555 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54546:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 21:26:46.751212 28555 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:26:46.765947 28722 tablet_service.cc:1467] Processing CreateTablet for tablet e04f7b90251c45a2be85aeeb7a5d4598 (DEFAULT_TABLE table=test-workload [id=cc08f1e1cdfd41ee842a6bad67b5382a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.766431 28722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04f7b90251c45a2be85aeeb7a5d4598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.767228 28647 tablet_service.cc:1467] Processing CreateTablet for tablet e04f7b90251c45a2be85aeeb7a5d4598 (DEFAULT_TABLE table=test-workload [id=cc08f1e1cdfd41ee842a6bad67b5382a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.767611 28647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04f7b90251c45a2be85aeeb7a5d4598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.767658 28797 tablet_service.cc:1467] Processing CreateTablet for tablet e04f7b90251c45a2be85aeeb7a5d4598 (DEFAULT_TABLE table=test-workload [id=cc08f1e1cdfd41ee842a6bad67b5382a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.768031 28797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04f7b90251c45a2be85aeeb7a5d4598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.774309 28856 tablet_bootstrap.cc:492] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9: Bootstrap starting.
I20240502 21:26:46.776074 28856 tablet_bootstrap.cc:654] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.776672 28855 tablet_bootstrap.cc:492] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca: Bootstrap starting.
I20240502 21:26:46.777691 28854 tablet_bootstrap.cc:492] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4: Bootstrap starting.
I20240502 21:26:46.778681 28855 tablet_bootstrap.cc:654] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.779367 28854 tablet_bootstrap.cc:654] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.780728 28856 tablet_bootstrap.cc:492] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9: No bootstrap required, opened a new log
I20240502 21:26:46.780855 28856 ts_tablet_manager.cc:1367] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 21:26:46.781222 28854 tablet_bootstrap.cc:492] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4: No bootstrap required, opened a new log
I20240502 21:26:46.781332 28855 tablet_bootstrap.cc:492] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca: No bootstrap required, opened a new log
I20240502 21:26:46.781337 28854 ts_tablet_manager.cc:1367] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:26:46.781457 28855 ts_tablet_manager.cc:1367] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 21:26:46.781597 28856 raft_consensus.cc:348] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 44255 } } peers { permanent_uuid: "e460e5c286b54472994554cc72c536ca" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35097 } } peers { permanent_uuid: "1facdf918c044d84a9f42c58520282d4" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 45755 } }
I20240502 21:26:46.781810 28856 raft_consensus.cc:374] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.781914 28856 raft_consensus.cc:729] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fedd5f8f9434e8691d930d5cb5c3bd9, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.781983 28854 raft_consensus.cc:348] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 44255 } } peers { permanent_uuid: "e460e5c286b54472994554cc72c536ca" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35097 } } peers { permanent_uuid: "1facdf918c044d84a9f42c58520282d4" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 45755 } }
I20240502 21:26:46.782105 28855 raft_consensus.cc:348] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 44255 } } peers { permanent_uuid: "e460e5c286b54472994554cc72c536ca" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35097 } } peers { permanent_uuid: "1facdf918c044d84a9f42c58520282d4" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 45755 } }
I20240502 21:26:46.782203 28854 raft_consensus.cc:374] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.782161 28856 consensus_queue.cc:260] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 44255 } } peers { permanent_uuid: "e460e5c286b54472994554cc72c536ca" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35097 } } peers { permanent_uuid: "1facdf918c044d84a9f42c58520282d4" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 45755 } }
I20240502 21:26:46.782316 28855 raft_consensus.cc:374] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.782341 28854 raft_consensus.cc:729] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1facdf918c044d84a9f42c58520282d4, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.782393 28855 raft_consensus.cc:729] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e460e5c286b54472994554cc72c536ca, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.782544 28854 consensus_queue.cc:260] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 44255 } } peers { permanent_uuid: "e460e5c286b54472994554cc72c536ca" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35097 } } peers { permanent_uuid: "1facdf918c044d84a9f42c58520282d4" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 45755 } }
I20240502 21:26:46.782687 28855 consensus_queue.cc:260] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 44255 } } peers { permanent_uuid: "e460e5c286b54472994554cc72c536ca" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35097 } } peers { permanent_uuid: "1facdf918c044d84a9f42c58520282d4" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 45755 } }
I20240502 21:26:46.784137 28854 ts_tablet_manager.cc:1398] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:46.788717 28856 ts_tablet_manager.cc:1398] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9: Time spent starting tablet: real 0.008s user 0.002s sys 0.006s
I20240502 21:26:46.789511 28855 ts_tablet_manager.cc:1398] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca: Time spent starting tablet: real 0.008s user 0.003s sys 0.005s
I20240502 21:26:46.835832 28862 raft_consensus.cc:482] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:46.835999 28862 raft_consensus.cc:504] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 44255 } } peers { permanent_uuid: "e460e5c286b54472994554cc72c536ca" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35097 } } peers { permanent_uuid: "1facdf918c044d84a9f42c58520282d4" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 45755 } }
I20240502 21:26:46.837100 28862 leader_election.cc:288] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fedd5f8f9434e8691d930d5cb5c3bd9 (127.25.16.65:44255), 1facdf918c044d84a9f42c58520282d4 (127.25.16.67:45755)
I20240502 21:26:46.842875 28657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04f7b90251c45a2be85aeeb7a5d4598" candidate_uuid: "e460e5c286b54472994554cc72c536ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9" is_pre_election: true
I20240502 21:26:46.843120 28657 raft_consensus.cc:2454] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e460e5c286b54472994554cc72c536ca in term 0.
I20240502 21:26:46.843068 28807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04f7b90251c45a2be85aeeb7a5d4598" candidate_uuid: "e460e5c286b54472994554cc72c536ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1facdf918c044d84a9f42c58520282d4" is_pre_election: true
I20240502 21:26:46.843317 28807 raft_consensus.cc:2454] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e460e5c286b54472994554cc72c536ca in term 0.
I20240502 21:26:46.843592 28697 leader_election.cc:302] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fedd5f8f9434e8691d930d5cb5c3bd9, e460e5c286b54472994554cc72c536ca; no voters:
I20240502 21:26:46.843873 28862 raft_consensus.cc:2781] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:46.843964 28862 raft_consensus.cc:482] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:46.844054 28862 raft_consensus.cc:3037] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.845883 28862 raft_consensus.cc:504] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 44255 } } peers { permanent_uuid: "e460e5c286b54472994554cc72c536ca" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35097 } } peers { permanent_uuid: "1facdf918c044d84a9f42c58520282d4" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 45755 } }
I20240502 21:26:46.846354 28862 leader_election.cc:288] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [CANDIDATE]: Term 1 election: Requested vote from peers 1fedd5f8f9434e8691d930d5cb5c3bd9 (127.25.16.65:44255), 1facdf918c044d84a9f42c58520282d4 (127.25.16.67:45755)
I20240502 21:26:46.846699 28657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04f7b90251c45a2be85aeeb7a5d4598" candidate_uuid: "e460e5c286b54472994554cc72c536ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9"
I20240502 21:26:46.846782 28807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04f7b90251c45a2be85aeeb7a5d4598" candidate_uuid: "e460e5c286b54472994554cc72c536ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1facdf918c044d84a9f42c58520282d4"
I20240502 21:26:46.846856 28657 raft_consensus.cc:3037] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.846969 28807 raft_consensus.cc:3037] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.848827 28657 raft_consensus.cc:2454] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e460e5c286b54472994554cc72c536ca in term 1.
I20240502 21:26:46.848910 28807 raft_consensus.cc:2454] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e460e5c286b54472994554cc72c536ca in term 1.
I20240502 21:26:46.849179 28697 leader_election.cc:302] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fedd5f8f9434e8691d930d5cb5c3bd9, e460e5c286b54472994554cc72c536ca; no voters:
I20240502 21:26:46.849424 28862 raft_consensus.cc:2781] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:46.849757 28862 raft_consensus.cc:686] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 1 LEADER]: Becoming Leader. State: Replica: e460e5c286b54472994554cc72c536ca, State: Running, Role: LEADER
I20240502 21:26:46.849994 28862 consensus_queue.cc:237] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 44255 } } peers { permanent_uuid: "e460e5c286b54472994554cc72c536ca" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35097 } } peers { permanent_uuid: "1facdf918c044d84a9f42c58520282d4" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 45755 } }
I20240502 21:26:46.852320 28555 catalog_manager.cc:5484] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca reported cstate change: term changed from 0 to 1, leader changed from <none> to e460e5c286b54472994554cc72c536ca (127.25.16.66). New cstate: current_term: 1 leader_uuid: "e460e5c286b54472994554cc72c536ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 44255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e460e5c286b54472994554cc72c536ca" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 35097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1facdf918c044d84a9f42c58520282d4" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 45755 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:47.309324 28862 consensus_queue.cc:1035] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1facdf918c044d84a9f42c58520282d4" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 45755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.314944 28862 consensus_queue.cc:1035] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fedd5f8f9434e8691d930d5cb5c3bd9" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 44255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.510493 28603 placement_policy_util.cc:407] tablet e04f7b90251c45a2be85aeeb7a5d4598: detected majority of replicas (2 of 3) at location L0
I20240502 21:26:47.510720 28608 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:47.511147 28608 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:47.511252 28608 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:48.513271 28603 placement_policy_util.cc:407] tablet e04f7b90251c45a2be85aeeb7a5d4598: detected majority of replicas (2 of 3) at location L0
I20240502 21:26:49.511641 28608 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:49.512112 28608 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:49.512233 28608 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:49.515568 28603 placement_policy_util.cc:407] tablet e04f7b90251c45a2be85aeeb7a5d4598: detected majority of replicas (2 of 3) at location L0
I20240502 21:26:50.518075 28603 placement_policy_util.cc:407] tablet e04f7b90251c45a2be85aeeb7a5d4598: detected majority of replicas (2 of 3) at location L0
I20240502 21:26:51.512646 28608 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:51.513103 28608 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:51.513221 28608 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:51.520417 28603 placement_policy_util.cc:407] tablet e04f7b90251c45a2be85aeeb7a5d4598: detected majority of replicas (2 of 3) at location L0
I20240502 21:26:52.522845 28603 placement_policy_util.cc:407] tablet e04f7b90251c45a2be85aeeb7a5d4598: detected majority of replicas (2 of 3) at location L0
I20240502 21:26:53.513638 28608 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:53.514109 28608 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:53.514242 28608 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:53.525288 28603 placement_policy_util.cc:407] tablet e04f7b90251c45a2be85aeeb7a5d4598: detected majority of replicas (2 of 3) at location L0
I20240502 21:26:54.527727 28603 placement_policy_util.cc:407] tablet e04f7b90251c45a2be85aeeb7a5d4598: detected majority of replicas (2 of 3) at location L0
I20240502 21:26:55.514632 28608 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:55.515115 28608 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:55.515234 28608 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:55.530331 28603 placement_policy_util.cc:407] tablet e04f7b90251c45a2be85aeeb7a5d4598: detected majority of replicas (2 of 3) at location L0
I20240502 21:26:56.532698 28603 placement_policy_util.cc:407] tablet e04f7b90251c45a2be85aeeb7a5d4598: detected majority of replicas (2 of 3) at location L0
I20240502 21:26:57.515588 28608 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:26:57.516052 28608 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:26:57.516162 28608 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:26:57.535043 28603 placement_policy_util.cc:407] tablet e04f7b90251c45a2be85aeeb7a5d4598: detected majority of replicas (2 of 3) at location L0
I20240502 21:26:57.908721 25665 tablet_server.cc:178] TabletServer@127.25.16.65:0 shutting down...
I20240502 21:26:57.921401 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:57.921671 25665 tablet_replica.cc:330] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9: stopping tablet replica
I20240502 21:26:57.921849 25665 raft_consensus.cc:2229] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:57.922035 25665 raft_consensus.cc:2258] T e04f7b90251c45a2be85aeeb7a5d4598 P 1fedd5f8f9434e8691d930d5cb5c3bd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:57.935645 25665 tablet_server.cc:194] TabletServer@127.25.16.65:0 shutdown complete.
I20240502 21:26:57.938472 25665 tablet_server.cc:178] TabletServer@127.25.16.66:0 shutting down...
I20240502 21:26:57.945868 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:57.946094 25665 tablet_replica.cc:330] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca: stopping tablet replica
I20240502 21:26:57.946269 25665 raft_consensus.cc:2229] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:57.946571 25665 raft_consensus.cc:2258] T e04f7b90251c45a2be85aeeb7a5d4598 P e460e5c286b54472994554cc72c536ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:57.959668 25665 tablet_server.cc:194] TabletServer@127.25.16.66:0 shutdown complete.
I20240502 21:26:57.971515 25665 tablet_server.cc:178] TabletServer@127.25.16.67:0 shutting down...
I20240502 21:26:57.979249 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:26:57.979456 25665 tablet_replica.cc:330] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4: stopping tablet replica
I20240502 21:26:57.979620 25665 raft_consensus.cc:2229] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:57.979804 25665 raft_consensus.cc:2258] T e04f7b90251c45a2be85aeeb7a5d4598 P 1facdf918c044d84a9f42c58520282d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:57.992957 25665 tablet_server.cc:194] TabletServer@127.25.16.67:0 shutdown complete.
I20240502 21:26:57.995812 25665 master.cc:537] Master@127.25.16.126:40261 shutting down...
I20240502 21:26:58.002569 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:58.002785 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:58.002899 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P 16181cda5c49423aa7d39f5f444db8d8: stopping tablet replica
I20240502 21:26:58.015532 25665 master.cc:552] Master@127.25.16.126:40261 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12571 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240502 21:26:58.025434 25665 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.16.126:42491
I20240502 21:26:58.025871 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:58.028187 28879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:58.028715 28880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:58.029218 28882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:58.030071 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:58.030303 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:58.030352 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:58.030382 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685218030375 us; error 0 us; skew 500 ppm
I20240502 21:26:58.030485 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:58.036020 25665 webserver.cc:460] Webserver started at http://127.25.16.126:41741/ using document root <none> and password file <none>
I20240502 21:26:58.036197 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:58.036242 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:58.036358 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:58.036909 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/master-0-root/instance:
uuid: "481b5308c76d43678924338449b5a9f6"
format_stamp: "Formatted at 2024-05-02 21:26:58 on dist-test-slave-b524"
I20240502 21:26:58.039124 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:58.040498 28887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.040797 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:58.040899 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/master-0-root
uuid: "481b5308c76d43678924338449b5a9f6"
format_stamp: "Formatted at 2024-05-02 21:26:58 on dist-test-slave-b524"
I20240502 21:26:58.040989 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:58.049741 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:58.050155 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:58.059685 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.126:42491
I20240502 21:26:58.059710 28939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.126:42491 every 8 connection(s)
I20240502 21:26:58.060864 28940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.064818 28940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6: Bootstrap starting.
I20240502 21:26:58.065817 28940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.067113 28940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6: No bootstrap required, opened a new log
I20240502 21:26:58.067602 28940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481b5308c76d43678924338449b5a9f6" member_type: VOTER }
I20240502 21:26:58.067718 28940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.067756 28940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481b5308c76d43678924338449b5a9f6, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.067896 28940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481b5308c76d43678924338449b5a9f6" member_type: VOTER }
I20240502 21:26:58.067997 28940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:26:58.068035 28940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:26:58.068082 28940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.069741 28940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481b5308c76d43678924338449b5a9f6" member_type: VOTER }
I20240502 21:26:58.069912 28940 leader_election.cc:302] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 481b5308c76d43678924338449b5a9f6; no voters:
I20240502 21:26:58.070211 28940 leader_election.cc:288] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:26:58.070314 28943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:58.070684 28943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [term 1 LEADER]: Becoming Leader. State: Replica: 481b5308c76d43678924338449b5a9f6, State: Running, Role: LEADER
I20240502 21:26:58.070887 28943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481b5308c76d43678924338449b5a9f6" member_type: VOTER }
I20240502 21:26:58.070957 28940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:26:58.071487 28944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "481b5308c76d43678924338449b5a9f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481b5308c76d43678924338449b5a9f6" member_type: VOTER } }
I20240502 21:26:58.071538 28945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 481b5308c76d43678924338449b5a9f6. Latest consensus state: current_term: 1 leader_uuid: "481b5308c76d43678924338449b5a9f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481b5308c76d43678924338449b5a9f6" member_type: VOTER } }
I20240502 21:26:58.071764 28944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:58.071848 28945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:58.072917 28948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:26:58.074510 28948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:26:58.077626 28948 catalog_manager.cc:1347] Generated new cluster ID: bd42348451b04d96b047cbaf1a09078b
I20240502 21:26:58.077714 28948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:26:58.078336 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:26:58.087770 28948 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:26:58.088498 28948 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:26:58.095726 28948 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6: Generated new TSK 0
I20240502 21:26:58.095921 28948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:26:58.110561 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:58.113265 28961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:58.113689 28962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:58.114056 28964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:58.115056 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:58.115232 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:58.115274 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:58.115300 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685218115297 us; error 0 us; skew 500 ppm
I20240502 21:26:58.115398 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:58.116493 25665 webserver.cc:460] Webserver started at http://127.25.16.65:43207/ using document root <none> and password file <none>
I20240502 21:26:58.116691 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:58.116739 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:58.116822 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:58.117362 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-0-root/instance:
uuid: "94f26f2c271544729391496f0fa81a58"
format_stamp: "Formatted at 2024-05-02 21:26:58 on dist-test-slave-b524"
I20240502 21:26:58.119563 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:58.120986 28969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.121270 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:58.121353 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-0-root
uuid: "94f26f2c271544729391496f0fa81a58"
format_stamp: "Formatted at 2024-05-02 21:26:58 on dist-test-slave-b524"
I20240502 21:26:58.121452 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:58.138411 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:58.138806 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:58.139250 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:58.140007 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:58.140056 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.140118 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:58.140157 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.151170 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.65:33159
I20240502 21:26:58.151189 29031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.65:33159 every 8 connection(s)
I20240502 21:26:58.152587 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:58.155663 29038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:58.156656 29037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:58.157547 29040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:58.157985 25665 server_base.cc:1034] running on GCE node
I20240502 21:26:58.158226 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:26:58.158288 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:58.158330 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685218158326 us; error 0 us; skew 500 ppm
I20240502 21:26:58.158524 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:58.160424 25665 webserver.cc:460] Webserver started at http://127.25.16.66:33071/ using document root <none> and password file <none>
I20240502 21:26:58.160540 29032 heartbeater.cc:347] Connected to a master server at 127.25.16.126:42491
I20240502 21:26:58.160662 29032 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:58.160672 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:58.160755 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:58.160878 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:58.161042 29032 heartbeater.cc:511] Master 127.25.16.126:42491 requested a full tablet report, sending...
I20240502 21:26:58.161530 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-1-root/instance:
uuid: "9846ea932cb14dff98f20d67f1829791"
format_stamp: "Formatted at 2024-05-02 21:26:58 on dist-test-slave-b524"
I20240502 21:26:58.161826 28905 ts_manager.cc:194] Registered new tserver with Master: 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159)
I20240502 21:26:58.162757 28905 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56856
I20240502 21:26:58.163848 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:26:58.165239 29045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.165540 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:58.165639 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-1-root
uuid: "9846ea932cb14dff98f20d67f1829791"
format_stamp: "Formatted at 2024-05-02 21:26:58 on dist-test-slave-b524"
I20240502 21:26:58.165721 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:58.182665 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:58.183053 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:58.183480 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:58.184235 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:58.184324 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.184399 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:58.184438 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.195227 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.66:46109
I20240502 21:26:58.195255 29107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.66:46109 every 8 connection(s)
I20240502 21:26:58.196617 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:58.199272 29111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:58.199858 29112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:58.201269 25665 server_base.cc:1034] running on GCE node
W20240502 21:26:58.202301 29114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:58.202630 29108 heartbeater.cc:347] Connected to a master server at 127.25.16.126:42491
I20240502 21:26:58.202733 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 21:26:58.202741 29108 heartbeater.cc:464] Registering TS with master...
W20240502 21:26:58.202816 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:26:58.202857 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685218202852 us; error 0 us; skew 500 ppm
I20240502 21:26:58.202980 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:58.203202 29108 heartbeater.cc:511] Master 127.25.16.126:42491 requested a full tablet report, sending...
I20240502 21:26:58.204046 28905 ts_manager.cc:194] Registered new tserver with Master: 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:26:58.204154 25665 webserver.cc:460] Webserver started at http://127.25.16.67:32789/ using document root <none> and password file <none>
I20240502 21:26:58.204386 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:58.204440 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:58.204558 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:58.204936 28905 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56872
I20240502 21:26:58.205160 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-2-root/instance:
uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
format_stamp: "Formatted at 2024-05-02 21:26:58 on dist-test-slave-b524"
I20240502 21:26:58.207331 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:26:58.208731 29119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.209028 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:58.209123 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-2-root
uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
format_stamp: "Formatted at 2024-05-02 21:26:58 on dist-test-slave-b524"
I20240502 21:26:58.209213 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:58.221450 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:58.221868 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:58.222290 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:58.223150 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:58.223205 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.223270 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:58.223305 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:58.235365 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.67:41785
I20240502 21:26:58.235394 29182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.67:41785 every 8 connection(s)
I20240502 21:26:58.241153 29183 heartbeater.cc:347] Connected to a master server at 127.25.16.126:42491
I20240502 21:26:58.241272 29183 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:58.241567 29183 heartbeater.cc:511] Master 127.25.16.126:42491 requested a full tablet report, sending...
I20240502 21:26:58.242309 28905 ts_manager.cc:194] Registered new tserver with Master: 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:58.242914 25665 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006747447s
I20240502 21:26:58.243043 28905 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56882
I20240502 21:26:59.164034 29032 heartbeater.cc:503] Master 127.25.16.126:42491 was elected leader, sending a full tablet report...
I20240502 21:26:59.206032 29108 heartbeater.cc:503] Master 127.25.16.126:42491 was elected leader, sending a full tablet report...
I20240502 21:26:59.244139 29183 heartbeater.cc:503] Master 127.25.16.126:42491 was elected leader, sending a full tablet report...
I20240502 21:26:59.269835 25665 test_util.cc:268] Using random seed: 1130720452
I20240502 21:26:59.280889 28905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56894:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 21:26:59.281630 28905 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:26:59.302095 28996 tablet_service.cc:1467] Processing CreateTablet for tablet ea5a9bdae3f3459f915eaf3bdd3d346c (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 21:26:59.302697 28996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5a9bdae3f3459f915eaf3bdd3d346c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.302635 28997 tablet_service.cc:1467] Processing CreateTablet for tablet 5010a09a5243452dad7cd9b90535a229 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 21:26:59.303000 28993 tablet_service.cc:1467] Processing CreateTablet for tablet 31c63a7053fa45b69eba6d7f7e78139b (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 21:26:59.302660 28994 tablet_service.cc:1467] Processing CreateTablet for tablet ee4eba5acb8d4af986a00f937de0270e (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 21:26:59.303094 28997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5010a09a5243452dad7cd9b90535a229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.303655 28994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4eba5acb8d4af986a00f937de0270e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.303622 28992 tablet_service.cc:1467] Processing CreateTablet for tablet af41f02157c94f55b2bc5090f566592e (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 21:26:59.302160 28995 tablet_service.cc:1467] Processing CreateTablet for tablet ccd9ffc92c5c47d088c530e919f2ff77 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 21:26:59.304033 28992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af41f02157c94f55b2bc5090f566592e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.304204 28995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd9ffc92c5c47d088c530e919f2ff77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.309919 28993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c63a7053fa45b69eba6d7f7e78139b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.311456 29147 tablet_service.cc:1467] Processing CreateTablet for tablet 5010a09a5243452dad7cd9b90535a229 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 21:26:59.311805 29073 tablet_service.cc:1467] Processing CreateTablet for tablet 5010a09a5243452dad7cd9b90535a229 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240502 21:26:59.312166 29146 tablet_service.cc:1467] Processing CreateTablet for tablet ea5a9bdae3f3459f915eaf3bdd3d346c (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 21:26:59.312615 29068 tablet_service.cc:1467] Processing CreateTablet for tablet af41f02157c94f55b2bc5090f566592e (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 21:26:59.312886 29145 tablet_service.cc:1467] Processing CreateTablet for tablet ccd9ffc92c5c47d088c530e919f2ff77 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 21:26:59.313210 29071 tablet_service.cc:1467] Processing CreateTablet for tablet ccd9ffc92c5c47d088c530e919f2ff77 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240502 21:26:59.313289 29072 tablet_service.cc:1467] Processing CreateTablet for tablet ea5a9bdae3f3459f915eaf3bdd3d346c (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240502 21:26:59.313536 29069 tablet_service.cc:1467] Processing CreateTablet for tablet 31c63a7053fa45b69eba6d7f7e78139b (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 21:26:59.311811 29147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5010a09a5243452dad7cd9b90535a229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.313994 29070 tablet_service.cc:1467] Processing CreateTablet for tablet ee4eba5acb8d4af986a00f937de0270e (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 21:26:59.314208 29144 tablet_service.cc:1467] Processing CreateTablet for tablet ee4eba5acb8d4af986a00f937de0270e (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240502 21:26:59.314585 29144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4eba5acb8d4af986a00f937de0270e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.315832 29143 tablet_service.cc:1467] Processing CreateTablet for tablet 31c63a7053fa45b69eba6d7f7e78139b (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240502 21:26:59.315954 28991 tablet_service.cc:1467] Processing CreateTablet for tablet 93477dec2c574c49a0bc9a7f022ad7fa (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 21:26:59.316224 29143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c63a7053fa45b69eba6d7f7e78139b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.316383 28991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93477dec2c574c49a0bc9a7f022ad7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.320144 28990 tablet_service.cc:1467] Processing CreateTablet for tablet 54a27d02080d420382add870508aaad4 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 21:26:59.320624 28990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a27d02080d420382add870508aaad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.320706 29146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5a9bdae3f3459f915eaf3bdd3d346c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.321465 28988 tablet_service.cc:1467] Processing CreateTablet for tablet ab35657135e44445aee4aa6961f70c27 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 21:26:59.321929 28988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab35657135e44445aee4aa6961f70c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.324518 29145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd9ffc92c5c47d088c530e919f2ff77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.324627 28989 tablet_service.cc:1467] Processing CreateTablet for tablet 1de7fda467344968ab81f653b8501210 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 21:26:59.325116 28989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de7fda467344968ab81f653b8501210. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.312179 29073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5010a09a5243452dad7cd9b90535a229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.328596 29070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4eba5acb8d4af986a00f937de0270e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.328764 29072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5a9bdae3f3459f915eaf3bdd3d346c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.328971 29071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd9ffc92c5c47d088c530e919f2ff77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.332458 29068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af41f02157c94f55b2bc5090f566592e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.333034 29069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c63a7053fa45b69eba6d7f7e78139b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.338186 28997 tablet_service.cc:1467] Processing CreateTablet for tablet 6d08f3643db34c8ba4d4391b630be366 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 21:26:59.338578 28997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d08f3643db34c8ba4d4391b630be366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.347334 29067 tablet_service.cc:1467] Processing CreateTablet for tablet 93477dec2c574c49a0bc9a7f022ad7fa (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 21:26:59.347823 29145 tablet_service.cc:1467] Processing CreateTablet for tablet af41f02157c94f55b2bc5090f566592e (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240502 21:26:59.347800 28994 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce57dd3a92645839765ba1ce6609430 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 21:26:59.351914 29144 tablet_service.cc:1467] Processing CreateTablet for tablet 93477dec2c574c49a0bc9a7f022ad7fa (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240502 21:26:59.353833 29072 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce57dd3a92645839765ba1ce6609430 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 21:26:59.354269 29145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af41f02157c94f55b2bc5090f566592e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.355290 28994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce57dd3a92645839765ba1ce6609430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.359236 29144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93477dec2c574c49a0bc9a7f022ad7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.359910 29072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce57dd3a92645839765ba1ce6609430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.362131 29142 tablet_service.cc:1467] Processing CreateTablet for tablet 54a27d02080d420382add870508aaad4 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 21:26:59.365523 29067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93477dec2c574c49a0bc9a7f022ad7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.366583 29065 tablet_service.cc:1467] Processing CreateTablet for tablet 1de7fda467344968ab81f653b8501210 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 21:26:59.367096 29065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de7fda467344968ab81f653b8501210. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.372517 29066 tablet_service.cc:1467] Processing CreateTablet for tablet 54a27d02080d420382add870508aaad4 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240502 21:26:59.372572 29141 tablet_service.cc:1467] Processing CreateTablet for tablet 1de7fda467344968ab81f653b8501210 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240502 21:26:59.376554 29066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a27d02080d420382add870508aaad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.383064 29204 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58: Bootstrap starting.
I20240502 21:26:59.383152 29140 tablet_service.cc:1467] Processing CreateTablet for tablet ab35657135e44445aee4aa6961f70c27 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 21:26:59.383574 29139 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce57dd3a92645839765ba1ce6609430 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240502 21:26:59.389930 29142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a27d02080d420382add870508aaad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.389698 29203 tablet_bootstrap.cc:492] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Bootstrap starting.
I20240502 21:26:59.392576 29064 tablet_service.cc:1467] Processing CreateTablet for tablet ab35657135e44445aee4aa6961f70c27 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240502 21:26:59.392743 29205 tablet_bootstrap.cc:492] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791: Bootstrap starting.
I20240502 21:26:59.393563 29064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab35657135e44445aee4aa6961f70c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.393533 29138 tablet_service.cc:1467] Processing CreateTablet for tablet 6d08f3643db34c8ba4d4391b630be366 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 21:26:59.394774 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de7fda467344968ab81f653b8501210. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.394714 29071 tablet_service.cc:1467] Processing CreateTablet for tablet 6d08f3643db34c8ba4d4391b630be366 (DEFAULT_TABLE table=test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240502 21:26:59.404335 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab35657135e44445aee4aa6961f70c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.404708 29138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d08f3643db34c8ba4d4391b630be366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.408437 29139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce57dd3a92645839765ba1ce6609430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.413367 29071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d08f3643db34c8ba4d4391b630be366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.417212 29204 tablet_bootstrap.cc:654] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.420148 29205 tablet_bootstrap.cc:654] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.420148 29203 tablet_bootstrap.cc:654] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.421972 29204 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58: No bootstrap required, opened a new log
I20240502 21:26:59.422134 29204 ts_tablet_manager.cc:1367] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20240502 21:26:59.422222 29205 tablet_bootstrap.cc:492] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791: No bootstrap required, opened a new log
I20240502 21:26:59.422350 29205 ts_tablet_manager.cc:1367] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20240502 21:26:59.422864 29204 raft_consensus.cc:348] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.422952 29205 raft_consensus.cc:348] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.423100 29205 raft_consensus.cc:374] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.423100 29204 raft_consensus.cc:374] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.423274 29205 raft_consensus.cc:729] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.423307 29204 raft_consensus.cc:729] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.423604 29205 consensus_queue.cc:260] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.423605 29204 consensus_queue.cc:260] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.424422 29204 ts_tablet_manager.cc:1398] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:59.424547 29203 tablet_bootstrap.cc:492] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b: No bootstrap required, opened a new log
I20240502 21:26:59.424703 29204 tablet_bootstrap.cc:492] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58: Bootstrap starting.
I20240502 21:26:59.424714 29203 ts_tablet_manager.cc:1367] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.003s
I20240502 21:26:59.425462 29203 raft_consensus.cc:348] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } }
I20240502 21:26:59.425717 29203 raft_consensus.cc:374] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.425796 29203 raft_consensus.cc:729] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.426002 29203 consensus_queue.cc:260] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } }
I20240502 21:26:59.426301 29204 tablet_bootstrap.cc:654] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.431422 29205 ts_tablet_manager.cc:1398] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20240502 21:26:59.431710 29205 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791: Bootstrap starting.
I20240502 21:26:59.433346 29205 tablet_bootstrap.cc:654] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.434497 29204 tablet_bootstrap.cc:492] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58: No bootstrap required, opened a new log
I20240502 21:26:59.434634 29204 ts_tablet_manager.cc:1367] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240502 21:26:59.435189 29204 raft_consensus.cc:348] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.435397 29204 raft_consensus.cc:374] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.435427 29203 ts_tablet_manager.cc:1398] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20240502 21:26:59.435484 29204 raft_consensus.cc:729] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.435719 29203 tablet_bootstrap.cc:492] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Bootstrap starting.
I20240502 21:26:59.435770 29204 consensus_queue.cc:260] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.436316 29204 ts_tablet_manager.cc:1398] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:59.437242 29204 tablet_bootstrap.cc:492] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58: Bootstrap starting.
I20240502 21:26:59.438823 29204 tablet_bootstrap.cc:654] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.439806 29203 tablet_bootstrap.cc:654] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.441408 29204 tablet_bootstrap.cc:492] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58: No bootstrap required, opened a new log
I20240502 21:26:59.441553 29204 ts_tablet_manager.cc:1367] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240502 21:26:59.442159 29204 raft_consensus.cc:348] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } }
I20240502 21:26:59.442337 29204 raft_consensus.cc:374] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.442430 29204 raft_consensus.cc:729] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.442656 29204 consensus_queue.cc:260] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } }
I20240502 21:26:59.443097 29204 ts_tablet_manager.cc:1398] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:59.443367 29204 tablet_bootstrap.cc:492] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58: Bootstrap starting.
I20240502 21:26:59.444254 29205 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791: No bootstrap required, opened a new log
I20240502 21:26:59.444449 29205 ts_tablet_manager.cc:1367] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240502 21:26:59.444957 29204 tablet_bootstrap.cc:654] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.445173 29203 tablet_bootstrap.cc:492] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b: No bootstrap required, opened a new log
I20240502 21:26:59.445318 29203 ts_tablet_manager.cc:1367] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240502 21:26:59.445210 29205 raft_consensus.cc:348] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.445430 29205 raft_consensus.cc:374] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.445529 29205 raft_consensus.cc:729] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.445791 29205 consensus_queue.cc:260] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.446027 29203 raft_consensus.cc:348] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.446225 29203 raft_consensus.cc:374] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.446312 29203 raft_consensus.cc:729] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.446525 29203 consensus_queue.cc:260] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.448227 29205 ts_tablet_manager.cc:1398] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:59.448798 29203 ts_tablet_manager.cc:1398] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:59.449162 29205 tablet_bootstrap.cc:492] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791: Bootstrap starting.
I20240502 21:26:59.450420 29203 tablet_bootstrap.cc:492] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Bootstrap starting.
I20240502 21:26:59.450603 29205 tablet_bootstrap.cc:654] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.451961 29203 tablet_bootstrap.cc:654] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.452466 29205 tablet_bootstrap.cc:492] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791: No bootstrap required, opened a new log
I20240502 21:26:59.452628 29204 tablet_bootstrap.cc:492] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58: No bootstrap required, opened a new log
I20240502 21:26:59.452616 29205 ts_tablet_manager.cc:1367] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:59.452850 29204 ts_tablet_manager.cc:1367] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.002s
I20240502 21:26:59.453555 29204 raft_consensus.cc:348] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.453783 29204 raft_consensus.cc:374] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.453629 29205 raft_consensus.cc:348] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } }
I20240502 21:26:59.453907 29203 tablet_bootstrap.cc:492] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b: No bootstrap required, opened a new log
I20240502 21:26:59.453881 29204 raft_consensus.cc:729] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.453946 29205 raft_consensus.cc:374] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.454054 29205 raft_consensus.cc:729] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.454062 29203 ts_tablet_manager.cc:1367] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:59.454173 29204 consensus_queue.cc:260] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.454273 29205 consensus_queue.cc:260] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } }
I20240502 21:26:59.454758 29205 ts_tablet_manager.cc:1398] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:59.455039 29205 tablet_bootstrap.cc:492] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791: Bootstrap starting.
I20240502 21:26:59.455170 29204 ts_tablet_manager.cc:1398] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:59.455451 29204 tablet_bootstrap.cc:492] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58: Bootstrap starting.
I20240502 21:26:59.456583 29205 tablet_bootstrap.cc:654] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.457022 29204 tablet_bootstrap.cc:654] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.454828 29203 raft_consensus.cc:348] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.458197 29203 raft_consensus.cc:374] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.458287 29203 raft_consensus.cc:729] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.458580 29203 consensus_queue.cc:260] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.466455 29205 tablet_bootstrap.cc:492] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791: No bootstrap required, opened a new log
I20240502 21:26:59.466573 29205 ts_tablet_manager.cc:1367] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240502 21:26:59.466589 29204 tablet_bootstrap.cc:492] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58: No bootstrap required, opened a new log
I20240502 21:26:59.466759 29204 ts_tablet_manager.cc:1367] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.002s
I20240502 21:26:59.467123 29205 raft_consensus.cc:348] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.467247 29205 raft_consensus.cc:374] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.467301 29205 raft_consensus.cc:729] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.467321 29204 raft_consensus.cc:348] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.467483 29204 raft_consensus.cc:374] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.467558 29204 raft_consensus.cc:729] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.467487 29205 consensus_queue.cc:260] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.467828 29203 ts_tablet_manager.cc:1398] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent starting tablet: real 0.014s user 0.005s sys 0.005s
I20240502 21:26:59.467952 29205 ts_tablet_manager.cc:1398] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 21:26:59.467841 29204 consensus_queue.cc:260] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.468060 29203 tablet_bootstrap.cc:492] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: Bootstrap starting.
I20240502 21:26:59.468181 29205 tablet_bootstrap.cc:492] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791: Bootstrap starting.
I20240502 21:26:59.468333 29204 ts_tablet_manager.cc:1398] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:59.469022 29204 tablet_bootstrap.cc:492] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58: Bootstrap starting.
I20240502 21:26:59.469458 29203 tablet_bootstrap.cc:654] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.469482 29205 tablet_bootstrap.cc:654] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.470244 29204 tablet_bootstrap.cc:654] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.476994 29203 tablet_bootstrap.cc:492] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: No bootstrap required, opened a new log
I20240502 21:26:59.477113 29203 ts_tablet_manager.cc:1367] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 21:26:59.477741 29204 tablet_bootstrap.cc:492] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58: No bootstrap required, opened a new log
I20240502 21:26:59.477677 29203 raft_consensus.cc:348] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.477830 29203 raft_consensus.cc:374] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.477844 29204 ts_tablet_manager.cc:1367] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20240502 21:26:59.477880 29203 raft_consensus.cc:729] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.478096 29203 consensus_queue.cc:260] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.478401 29204 raft_consensus.cc:348] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.478564 29204 raft_consensus.cc:374] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.478614 29204 raft_consensus.cc:729] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.478817 29204 consensus_queue.cc:260] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.479115 29203 ts_tablet_manager.cc:1398] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:59.479233 29204 ts_tablet_manager.cc:1398] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 21:26:59.479353 29203 tablet_bootstrap.cc:492] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Bootstrap starting.
I20240502 21:26:59.479434 29204 tablet_bootstrap.cc:492] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58: Bootstrap starting.
I20240502 21:26:59.479595 29205 tablet_bootstrap.cc:492] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791: No bootstrap required, opened a new log
I20240502 21:26:59.479759 29205 ts_tablet_manager.cc:1367] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240502 21:26:59.480494 29203 tablet_bootstrap.cc:654] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.480494 29204 tablet_bootstrap.cc:654] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.480423 29205 raft_consensus.cc:348] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.480720 29205 raft_consensus.cc:374] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.480835 29205 raft_consensus.cc:729] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.481065 29205 consensus_queue.cc:260] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.481654 29205 ts_tablet_manager.cc:1398] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:59.481961 29205 tablet_bootstrap.cc:492] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791: Bootstrap starting.
I20240502 21:26:59.482981 29210 raft_consensus.cc:482] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.483115 29210 raft_consensus.cc:504] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } }
I20240502 21:26:59.483363 29205 tablet_bootstrap.cc:654] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.484009 29210 leader_election.cc:288] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:26:59.484453 29203 tablet_bootstrap.cc:492] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b: No bootstrap required, opened a new log
I20240502 21:26:59.484580 29203 ts_tablet_manager.cc:1367] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20240502 21:26:59.485164 29203 raft_consensus.cc:348] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.485343 29203 raft_consensus.cc:374] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.485399 29203 raft_consensus.cc:729] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.485605 29203 consensus_queue.cc:260] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.486119 29203 ts_tablet_manager.cc:1398] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:59.486357 29203 tablet_bootstrap.cc:492] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b: Bootstrap starting.
I20240502 21:26:59.487519 29203 tablet_bootstrap.cc:654] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.489233 29203 tablet_bootstrap.cc:492] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b: No bootstrap required, opened a new log
I20240502 21:26:59.489357 29203 ts_tablet_manager.cc:1367] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:59.489935 29203 raft_consensus.cc:348] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.490103 29203 raft_consensus.cc:374] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.490162 29203 raft_consensus.cc:729] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.490382 29203 consensus_queue.cc:260] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.490906 29203 ts_tablet_manager.cc:1398] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 21:26:59.491140 29203 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b: Bootstrap starting.
I20240502 21:26:59.492846 29203 tablet_bootstrap.cc:654] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.492880 29204 tablet_bootstrap.cc:492] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58: No bootstrap required, opened a new log
I20240502 21:26:59.492985 29204 ts_tablet_manager.cc:1367] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 21:26:59.493913 29204 raft_consensus.cc:348] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.494092 29204 raft_consensus.cc:374] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.494158 29204 raft_consensus.cc:729] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.494398 29204 consensus_queue.cc:260] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.496560 29205 tablet_bootstrap.cc:492] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791: No bootstrap required, opened a new log
I20240502 21:26:59.496691 29205 ts_tablet_manager.cc:1367] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20240502 21:26:59.496932 29203 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b: No bootstrap required, opened a new log
I20240502 21:26:59.496966 29204 ts_tablet_manager.cc:1398] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:26:59.497054 29203 ts_tablet_manager.cc:1367] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:26:59.497287 29205 raft_consensus.cc:348] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.497577 29205 raft_consensus.cc:374] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.497637 29205 raft_consensus.cc:729] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.497865 29205 consensus_queue.cc:260] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.497835 29203 raft_consensus.cc:348] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.498243 29203 raft_consensus.cc:374] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.498302 29203 raft_consensus.cc:729] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.498485 29203 consensus_queue.cc:260] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.498945 29203 ts_tablet_manager.cc:1398] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:59.499176 29203 tablet_bootstrap.cc:492] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Bootstrap starting.
I20240502 21:26:59.499491 29204 tablet_bootstrap.cc:492] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: Bootstrap starting.
I20240502 21:26:59.500409 29203 tablet_bootstrap.cc:654] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.500682 29204 tablet_bootstrap.cc:654] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.501341 29205 ts_tablet_manager.cc:1398] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:59.503639 29205 tablet_bootstrap.cc:492] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791: Bootstrap starting.
I20240502 21:26:59.504740 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5010a09a5243452dad7cd9b90535a229" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791" is_pre_election: true
I20240502 21:26:59.505035 29083 raft_consensus.cc:2454] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 0.
I20240502 21:26:59.505409 28972 leader_election.cc:302] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.505681 29210 raft_consensus.cc:2781] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.505769 29210 raft_consensus.cc:482] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.505841 29210 raft_consensus.cc:3037] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.506146 29204 tablet_bootstrap.cc:492] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: No bootstrap required, opened a new log
I20240502 21:26:59.506265 29204 ts_tablet_manager.cc:1367] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240502 21:26:59.506672 29205 tablet_bootstrap.cc:654] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.506847 29204 raft_consensus.cc:348] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.508248 29210 raft_consensus.cc:504] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } }
I20240502 21:26:59.508375 29204 raft_consensus.cc:374] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.508430 29204 raft_consensus.cc:729] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.508514 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5010a09a5243452dad7cd9b90535a229" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" is_pre_election: true
I20240502 21:26:59.508622 29204 consensus_queue.cc:260] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.508775 29157 raft_consensus.cc:2454] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 0.
I20240502 21:26:59.508849 29210 leader_election.cc:288] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 election: Requested vote from peers 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:26:59.507529 29203 tablet_bootstrap.cc:492] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b: No bootstrap required, opened a new log
I20240502 21:26:59.509023 29203 ts_tablet_manager.cc:1367] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:26:59.509406 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5010a09a5243452dad7cd9b90535a229" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:26:59.509565 29157 raft_consensus.cc:3037] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.509593 29203 raft_consensus.cc:348] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.509676 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5010a09a5243452dad7cd9b90535a229" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:26:59.509783 29203 raft_consensus.cc:374] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.509851 29203 raft_consensus.cc:729] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.510061 29203 consensus_queue.cc:260] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.509852 29083 raft_consensus.cc:3037] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.510924 29211 raft_consensus.cc:482] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.511039 29211 raft_consensus.cc:504] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.511855 29211 leader_election.cc:288] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:26:59.513413 29083 raft_consensus.cc:2454] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 1.
I20240502 21:26:59.514056 28972 leader_election.cc:302] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.514521 29157 raft_consensus.cc:2454] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 1.
I20240502 21:26:59.515985 29210 raft_consensus.cc:2781] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.516105 29210 raft_consensus.cc:686] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:26:59.516674 29210 consensus_queue.cc:237] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } }
I20240502 21:26:59.520244 29209 raft_consensus.cc:482] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.520378 29209 raft_consensus.cc:504] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.520659 29204 ts_tablet_manager.cc:1398] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: Time spent starting tablet: real 0.014s user 0.004s sys 0.003s
I20240502 21:26:59.520992 29204 tablet_bootstrap.cc:492] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58: Bootstrap starting.
I20240502 21:26:59.521232 29209 leader_election.cc:288] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.522039 29210 raft_consensus.cc:482] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.522166 29210 raft_consensus.cc:504] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.523187 29210 leader_election.cc:288] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.525868 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791" is_pre_election: true
I20240502 21:26:59.526084 29083 raft_consensus.cc:2454] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 0.
I20240502 21:26:59.526804 29204 tablet_bootstrap.cc:654] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.527213 28972 leader_election.cc:302] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.528172 29210 raft_consensus.cc:2781] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.528249 29210 raft_consensus.cc:482] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.528340 29210 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.528018 28904 catalog_manager.cc:5484] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 1 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.532462 29203 ts_tablet_manager.cc:1398] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20240502 21:26:59.532532 29204 tablet_bootstrap.cc:492] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58: No bootstrap required, opened a new log
I20240502 21:26:59.532645 29204 ts_tablet_manager.cc:1367] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 21:26:59.532842 29233 raft_consensus.cc:482] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.532950 29233 raft_consensus.cc:504] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.533313 29204 raft_consensus.cc:348] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.533473 29204 raft_consensus.cc:374] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.533524 29204 raft_consensus.cc:729] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.533704 29204 consensus_queue.cc:260] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.533857 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791" is_pre_election: true
I20240502 21:26:59.534148 29204 ts_tablet_manager.cc:1398] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 21:26:59.534278 28972 leader_election.cc:341] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.534376 29204 tablet_bootstrap.cc:492] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58: Bootstrap starting.
I20240502 21:26:59.535413 29203 tablet_bootstrap.cc:492] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Bootstrap starting.
I20240502 21:26:59.535729 29204 tablet_bootstrap.cc:654] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.537318 29233 leader_election.cc:288] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.538136 29203 tablet_bootstrap.cc:654] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.543351 29205 tablet_bootstrap.cc:492] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791: No bootstrap required, opened a new log
I20240502 21:26:59.543489 29205 ts_tablet_manager.cc:1367] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.006s
I20240502 21:26:59.544201 29205 raft_consensus.cc:348] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.544413 29205 raft_consensus.cc:374] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.544571 29205 raft_consensus.cc:729] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.544790 29205 consensus_queue.cc:260] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.545295 29205 ts_tablet_manager.cc:1398] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:59.548981 29210 raft_consensus.cc:504] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.549144 29205 tablet_bootstrap.cc:492] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791: Bootstrap starting.
I20240502 21:26:59.549849 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:26:59.550047 29083 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.550436 29205 tablet_bootstrap.cc:654] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.552865 29083 raft_consensus.cc:2454] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 1.
I20240502 21:26:59.553225 28972 leader_election.cc:302] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.555227 29210 leader_election.cc:288] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.555377 29210 raft_consensus.cc:2781] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.555473 29210 raft_consensus.cc:686] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:26:59.555709 29210 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.556759 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce57dd3a92645839765ba1ce6609430" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58" is_pre_election: true
W20240502 21:26:59.557396 29047 leader_election.cc:341] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.558679 29209 raft_consensus.cc:482] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.558808 29209 raft_consensus.cc:504] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.559363 29209 leader_election.cc:288] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.559830 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd9ffc92c5c47d088c530e919f2ff77" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58" is_pre_election: true
I20240502 21:26:59.560014 29007 raft_consensus.cc:2454] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 0.
I20240502 21:26:59.560366 29047 leader_election.cc:302] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.560608 29209 raft_consensus.cc:2781] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.560674 29209 raft_consensus.cc:482] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.560729 29209 raft_consensus.cc:3037] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.562471 29247 raft_consensus.cc:482] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.562583 29247 raft_consensus.cc:504] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.563336 29209 raft_consensus.cc:504] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.563541 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4eba5acb8d4af986a00f937de0270e" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58" is_pre_election: true
I20240502 21:26:59.563702 29007 raft_consensus.cc:2454] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 0.
I20240502 21:26:59.563884 29209 leader_election.cc:288] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.564241 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd9ffc92c5c47d088c530e919f2ff77" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:26:59.564424 29007 raft_consensus.cc:3037] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.565390 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" is_pre_election: true
I20240502 21:26:59.565570 29157 raft_consensus.cc:2454] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 0.
I20240502 21:26:59.565966 29203 tablet_bootstrap.cc:492] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b: No bootstrap required, opened a new log
I20240502 21:26:59.566089 29203 ts_tablet_manager.cc:1367] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240502 21:26:59.566131 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" is_pre_election: true
I20240502 21:26:59.566329 29157 raft_consensus.cc:2454] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 0.
I20240502 21:26:59.566764 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:26:59.568974 29007 raft_consensus.cc:2454] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 1.
I20240502 21:26:59.569582 28972 leader_election.cc:302] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters: 9846ea932cb14dff98f20d67f1829791
I20240502 21:26:59.568490 29203 raft_consensus.cc:348] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.570031 29233 raft_consensus.cc:2781] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.570578 29157 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.571251 29247 leader_election.cc:288] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.572718 29047 leader_election.cc:302] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.572970 29247 raft_consensus.cc:2781] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.573033 29047 leader_election.cc:302] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.573078 29247 raft_consensus.cc:482] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.573168 29247 raft_consensus.cc:3037] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.573958 29209 raft_consensus.cc:2781] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.574141 29233 raft_consensus.cc:482] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.574290 29203 raft_consensus.cc:374] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.581008 29203 raft_consensus.cc:729] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.581957 29233 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.582518 29203 consensus_queue.cc:260] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.575888 28903 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 1 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.584937 29203 ts_tablet_manager.cc:1398] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 21:26:59.585552 29205 tablet_bootstrap.cc:492] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791: No bootstrap required, opened a new log
I20240502 21:26:59.586041 29209 raft_consensus.cc:686] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Becoming Leader. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:26:59.586165 29203 tablet_bootstrap.cc:492] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Bootstrap starting.
I20240502 21:26:59.586582 29205 ts_tablet_manager.cc:1367] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.008s
I20240502 21:26:59.587359 29157 raft_consensus.cc:2454] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 1.
I20240502 21:26:59.587311 29205 raft_consensus.cc:348] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.587476 29203 tablet_bootstrap.cc:654] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.587505 29205 raft_consensus.cc:374] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.587579 29205 raft_consensus.cc:729] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.587810 29205 consensus_queue.cc:260] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.589766 29205 ts_tablet_manager.cc:1398] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:59.591655 29204 tablet_bootstrap.cc:492] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58: No bootstrap required, opened a new log
I20240502 21:26:59.591804 29204 ts_tablet_manager.cc:1367] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58: Time spent bootstrapping tablet: real 0.057s user 0.015s sys 0.022s
I20240502 21:26:59.592514 29204 raft_consensus.cc:348] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.592702 29204 raft_consensus.cc:374] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.592794 29204 raft_consensus.cc:729] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.592799 29247 raft_consensus.cc:504] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.593029 29204 consensus_queue.cc:260] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.593539 29204 ts_tablet_manager.cc:1398] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:59.593823 29204 tablet_bootstrap.cc:492] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58: Bootstrap starting.
I20240502 21:26:59.593788 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4eba5acb8d4af986a00f937de0270e" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:26:59.594798 29247 leader_election.cc:288] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.594795 29007 raft_consensus.cc:3037] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.596352 29205 tablet_bootstrap.cc:492] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791: Bootstrap starting.
I20240502 21:26:59.596698 29237 raft_consensus.cc:482] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.597651 29204 tablet_bootstrap.cc:654] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.589213 29209 consensus_queue.cc:237] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.600134 29237 raft_consensus.cc:504] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.602145 29233 raft_consensus.cc:504] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.602303 29007 raft_consensus.cc:2454] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 1.
I20240502 21:26:59.604142 29047 leader_election.cc:302] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.604429 29247 raft_consensus.cc:2781] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.605211 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:26:59.605531 29204 tablet_bootstrap.cc:492] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58: No bootstrap required, opened a new log
I20240502 21:26:59.605695 29233 leader_election.cc:288] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.605737 29204 ts_tablet_manager.cc:1367] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240502 21:26:59.606056 29237 leader_election.cc:288] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
W20240502 21:26:59.606655 28972 leader_election.cc:341] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.606606 29204 raft_consensus.cc:348] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.606909 29204 raft_consensus.cc:374] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.607039 29204 raft_consensus.cc:729] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.607059 29247 raft_consensus.cc:686] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Becoming Leader. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:26:59.607290 29205 tablet_bootstrap.cc:654] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.607713 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:26:59.607784 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791" is_pre_election: true
I20240502 21:26:59.609318 29157 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.609584 29204 consensus_queue.cc:260] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.609946 29203 tablet_bootstrap.cc:492] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b: No bootstrap required, opened a new log
I20240502 21:26:59.610926 29204 ts_tablet_manager.cc:1398] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20240502 21:26:59.611183 28972 leader_election.cc:341] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.607733 29156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" is_pre_election: true
I20240502 21:26:59.611768 29204 tablet_bootstrap.cc:492] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58: Bootstrap starting.
I20240502 21:26:59.607707 29247 consensus_queue.cc:237] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.613186 29203 ts_tablet_manager.cc:1367] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.007s
W20240502 21:26:59.613832 28972 leader_election.cc:341] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.613801 29156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce57dd3a92645839765ba1ce6609430" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" is_pre_election: true
I20240502 21:26:59.613976 28972 leader_election.cc:302] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94f26f2c271544729391496f0fa81a58; no voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791
I20240502 21:26:59.614459 29237 raft_consensus.cc:2726] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 21:26:59.615382 29048 leader_election.cc:341] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.616155 29203 raft_consensus.cc:348] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.616454 29203 raft_consensus.cc:374] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.616433 29048 leader_election.cc:302] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9846ea932cb14dff98f20d67f1829791; no voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58
I20240502 21:26:59.614634 29155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd9ffc92c5c47d088c530e919f2ff77" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" is_pre_election: true
I20240502 21:26:59.616727 29155 raft_consensus.cc:2454] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 0.
I20240502 21:26:59.614791 29152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4eba5acb8d4af986a00f937de0270e" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:26:59.618369 29152 raft_consensus.cc:3037] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.618523 29209 raft_consensus.cc:2726] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:59.618639 29204 tablet_bootstrap.cc:654] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.618765 29203 raft_consensus.cc:729] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.614709 29153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4eba5acb8d4af986a00f937de0270e" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" is_pre_election: true
I20240502 21:26:59.614759 29154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd9ffc92c5c47d088c530e919f2ff77" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:26:59.620311 29203 consensus_queue.cc:260] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.621567 29154 raft_consensus.cc:3037] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.625874 29203 ts_tablet_manager.cc:1398] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240502 21:26:59.626813 29157 raft_consensus.cc:2454] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 1.
I20240502 21:26:59.621909 28903 catalog_manager.cc:5484] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9846ea932cb14dff98f20d67f1829791 (127.25.16.66). New cstate: current_term: 1 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.627977 29203 tablet_bootstrap.cc:492] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Bootstrap starting.
I20240502 21:26:59.630628 28972 leader_election.cc:302] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters: 9846ea932cb14dff98f20d67f1829791
I20240502 21:26:59.632454 29237 raft_consensus.cc:2781] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.632647 29237 raft_consensus.cc:686] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:26:59.634387 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93477dec2c574c49a0bc9a7f022ad7fa" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58" is_pre_election: true
I20240502 21:26:59.634301 29237 consensus_queue.cc:237] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.634935 29007 raft_consensus.cc:2454] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 0.
I20240502 21:26:59.636394 29121 leader_election.cc:302] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:26:59.637266 29211 raft_consensus.cc:2781] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.637893 29211 raft_consensus.cc:482] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.638037 29211 raft_consensus.cc:3037] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.639556 29154 raft_consensus.cc:2454] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 1.
I20240502 21:26:59.638175 28903 catalog_manager.cc:5484] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9846ea932cb14dff98f20d67f1829791 (127.25.16.66). New cstate: current_term: 1 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.642570 29152 raft_consensus.cc:2454] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 1.
I20240502 21:26:59.643862 28905 catalog_manager.cc:5484] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 1 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.644554 29211 raft_consensus.cc:504] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.644943 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93477dec2c574c49a0bc9a7f022ad7fa" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791" is_pre_election: true
I20240502 21:26:59.645169 29083 raft_consensus.cc:2454] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 0.
I20240502 21:26:59.645637 29211 leader_election.cc:288] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:26:59.645678 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93477dec2c574c49a0bc9a7f022ad7fa" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:26:59.645869 29083 raft_consensus.cc:3037] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.648185 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93477dec2c574c49a0bc9a7f022ad7fa" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:26:59.648396 29007 raft_consensus.cc:3037] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.650688 29083 raft_consensus.cc:2454] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 1.
I20240502 21:26:59.650871 29211 raft_consensus.cc:482] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.651014 29211 raft_consensus.cc:504] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.651129 29122 leader_election.cc:302] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.651901 29211 leader_election.cc:288] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:26:59.652140 29211 raft_consensus.cc:2781] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.657174 29007 raft_consensus.cc:2454] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 1.
I20240502 21:26:59.658308 29211 raft_consensus.cc:686] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:26:59.658679 29211 consensus_queue.cc:237] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.661121 29203 tablet_bootstrap.cc:654] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.661355 29205 tablet_bootstrap.cc:492] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791: No bootstrap required, opened a new log
I20240502 21:26:59.651968 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58" is_pre_election: true
I20240502 21:26:59.665022 29006 raft_consensus.cc:2454] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 0.
I20240502 21:26:59.665346 29205 ts_tablet_manager.cc:1367] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.004s
I20240502 21:26:59.666113 29205 raft_consensus.cc:348] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.666330 29205 raft_consensus.cc:374] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.666417 29205 raft_consensus.cc:729] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.666667 29205 consensus_queue.cc:260] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.667248 29205 ts_tablet_manager.cc:1398] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:59.667529 29205 tablet_bootstrap.cc:492] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791: Bootstrap starting.
I20240502 21:26:59.669998 29256 raft_consensus.cc:482] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.670156 29256 raft_consensus.cc:504] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.671119 29256 leader_election.cc:288] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:26:59.671773 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce57dd3a92645839765ba1ce6609430" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58" is_pre_election: true
I20240502 21:26:59.671962 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791" is_pre_election: true
I20240502 21:26:59.672559 29205 tablet_bootstrap.cc:654] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.672508 29121 leader_election.cc:302] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:26:59.672752 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce57dd3a92645839765ba1ce6609430" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791" is_pre_election: true
W20240502 21:26:59.672868 29121 leader_election.cc:341] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.672981 29083 raft_consensus.cc:2454] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 0.
I20240502 21:26:59.673102 29257 raft_consensus.cc:2781] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.673215 29257 raft_consensus.cc:482] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.673316 29257 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:26:59.673394 29122 leader_election.cc:341] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.673755 29122 leader_election.cc:302] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791; no voters: 94f26f2c271544729391496f0fa81a58
I20240502 21:26:59.674181 29256 raft_consensus.cc:2781] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.674283 29256 raft_consensus.cc:482] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.674387 29256 raft_consensus.cc:3037] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.675225 29204 tablet_bootstrap.cc:492] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58: No bootstrap required, opened a new log
I20240502 21:26:59.675392 29204 ts_tablet_manager.cc:1367] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.013s
I20240502 21:26:59.676548 29257 raft_consensus.cc:504] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.677402 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:26:59.677587 29006 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.677639 29256 raft_consensus.cc:504] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.676126 29204 raft_consensus.cc:348] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.678050 29204 raft_consensus.cc:374] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.678154 29204 raft_consensus.cc:729] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.678377 29256 leader_election.cc:288] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:26:59.678915 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce57dd3a92645839765ba1ce6609430" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:26:59.679118 29083 raft_consensus.cc:3037] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.680642 29006 raft_consensus.cc:2454] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 1.
I20240502 21:26:59.681290 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce57dd3a92645839765ba1ce6609430" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:26:59.681607 29121 leader_election.cc:302] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:26:59.681979 29083 raft_consensus.cc:2454] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 1.
I20240502 21:26:59.682518 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:26:59.681967 29257 leader_election.cc:288] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:26:59.682876 29256 raft_consensus.cc:2781] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.682991 29122 leader_election.cc:302] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.683117 29256 raft_consensus.cc:686] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:26:59.683218 29205 tablet_bootstrap.cc:492] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791: No bootstrap required, opened a new log
W20240502 21:26:59.683286 29122 leader_election.cc:341] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.683372 29205 ts_tablet_manager.cc:1367] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.006s
I20240502 21:26:59.678378 29204 consensus_queue.cc:260] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.683765 29006 raft_consensus.cc:3037] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.683907 29211 raft_consensus.cc:2781] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.684070 29211 raft_consensus.cc:686] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:26:59.684129 29205 raft_consensus.cc:348] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.684376 29205 raft_consensus.cc:374] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.684197 28903 catalog_manager.cc:5484] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 1 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:59.684490 29211 consensus_queue.cc:237] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.684475 29205 raft_consensus.cc:729] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.683539 29256 consensus_queue.cc:237] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.685297 29205 consensus_queue.cc:260] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.686219 29205 ts_tablet_manager.cc:1398] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:59.686522 29205 tablet_bootstrap.cc:492] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791: Bootstrap starting.
I20240502 21:26:59.687098 29006 raft_consensus.cc:2454] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 1.
I20240502 21:26:59.688742 29204 ts_tablet_manager.cc:1398] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 21:26:59.689514 29203 tablet_bootstrap.cc:492] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b: No bootstrap required, opened a new log
I20240502 21:26:59.689677 29203 ts_tablet_manager.cc:1367] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.001s
I20240502 21:26:59.690264 29205 tablet_bootstrap.cc:654] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.690320 29203 raft_consensus.cc:348] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.690522 29203 raft_consensus.cc:374] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.690606 29203 raft_consensus.cc:729] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.690853 29203 consensus_queue.cc:260] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.691357 29203 ts_tablet_manager.cc:1398] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:59.691615 29203 tablet_bootstrap.cc:492] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Bootstrap starting.
I20240502 21:26:59.693205 29203 tablet_bootstrap.cc:654] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.694311 28903 catalog_manager.cc:5484] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 1 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:59.694836 28903 catalog_manager.cc:5484] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 1 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:59.695034 29205 tablet_bootstrap.cc:492] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791: No bootstrap required, opened a new log
I20240502 21:26:59.695240 29205 ts_tablet_manager.cc:1367] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20240502 21:26:59.695909 29205 raft_consensus.cc:348] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.696058 29205 raft_consensus.cc:374] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.696137 29205 raft_consensus.cc:729] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.696358 29205 consensus_queue.cc:260] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.696720 29205 ts_tablet_manager.cc:1398] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:59.696954 29205 tablet_bootstrap.cc:492] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791: Bootstrap starting.
I20240502 21:26:59.698431 29205 tablet_bootstrap.cc:654] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.698758 29203 tablet_bootstrap.cc:492] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b: No bootstrap required, opened a new log
I20240502 21:26:59.699046 29203 ts_tablet_manager.cc:1367] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240502 21:26:59.699661 29203 raft_consensus.cc:348] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.699867 29203 raft_consensus.cc:374] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.699928 29203 raft_consensus.cc:729] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.700095 29203 consensus_queue.cc:260] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.700318 29205 tablet_bootstrap.cc:492] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791: No bootstrap required, opened a new log
I20240502 21:26:59.700443 29205 ts_tablet_manager.cc:1367] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:26:59.700536 29203 ts_tablet_manager.cc:1398] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:59.701033 29205 raft_consensus.cc:348] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.701169 29205 raft_consensus.cc:374] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.701220 29205 raft_consensus.cc:729] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.701386 29205 consensus_queue.cc:260] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.701797 29205 ts_tablet_manager.cc:1398] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:59.712589 29247 raft_consensus.cc:482] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.712711 29247 raft_consensus.cc:504] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.713198 29247 leader_election.cc:288] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.713524 29152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de7fda467344968ab81f653b8501210" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" is_pre_election: true
I20240502 21:26:59.713523 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de7fda467344968ab81f653b8501210" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58" is_pre_election: true
I20240502 21:26:59.713765 29006 raft_consensus.cc:2454] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 0.
I20240502 21:26:59.713763 29152 raft_consensus.cc:2454] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 0.
I20240502 21:26:59.714138 29048 leader_election.cc:302] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.714380 29247 raft_consensus.cc:2781] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.714465 29247 raft_consensus.cc:482] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.714548 29247 raft_consensus.cc:3037] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.716185 29247 raft_consensus.cc:504] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.716686 29247 leader_election.cc:288] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.717016 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de7fda467344968ab81f653b8501210" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:26:59.717036 29152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de7fda467344968ab81f653b8501210" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:26:59.717180 29006 raft_consensus.cc:3037] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.717200 29152 raft_consensus.cc:3037] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.718986 29152 raft_consensus.cc:2454] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 1.
I20240502 21:26:59.718991 29006 raft_consensus.cc:2454] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 1.
I20240502 21:26:59.719301 29048 leader_election.cc:302] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.719543 29247 raft_consensus.cc:2781] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.719682 29247 raft_consensus.cc:686] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Becoming Leader. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:26:59.719933 29247 consensus_queue.cc:237] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.722110 28903 catalog_manager.cc:5484] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9846ea932cb14dff98f20d67f1829791 (127.25.16.66). New cstate: current_term: 1 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.745505 29237 raft_consensus.cc:482] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.745666 29237 raft_consensus.cc:504] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.746126 29237 leader_election.cc:288] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.746516 29152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" is_pre_election: true
I20240502 21:26:59.746519 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791" is_pre_election: true
I20240502 21:26:59.746753 29152 raft_consensus.cc:2454] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 0.
I20240502 21:26:59.746773 29083 raft_consensus.cc:2454] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 0.
I20240502 21:26:59.747172 28972 leader_election.cc:302] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.747408 29237 raft_consensus.cc:2781] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.747493 29237 raft_consensus.cc:482] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.747581 29237 raft_consensus.cc:3037] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.749460 29237 raft_consensus.cc:504] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.749930 29237 leader_election.cc:288] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.750275 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:26:59.750294 29152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:26:59.750437 29083 raft_consensus.cc:3037] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.750471 29152 raft_consensus.cc:3037] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.752219 29152 raft_consensus.cc:2454] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 1.
I20240502 21:26:59.752249 29083 raft_consensus.cc:2454] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 1.
I20240502 21:26:59.752574 28972 leader_election.cc:302] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:26:59.752799 29237 raft_consensus.cc:2781] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.752933 29237 raft_consensus.cc:686] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:26:59.753170 29237 consensus_queue.cc:237] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.755275 28903 catalog_manager.cc:5484] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 1 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.787003 29247 raft_consensus.cc:482] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.787128 29247 raft_consensus.cc:504] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.787631 29247 leader_election.cc:288] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.787988 29152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" is_pre_election: true
I20240502 21:26:59.787950 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58" is_pre_election: true
I20240502 21:26:59.788194 29152 raft_consensus.cc:2379] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9846ea932cb14dff98f20d67f1829791 in current term 1: Already voted for candidate 94f26f2c271544729391496f0fa81a58 in this term.
I20240502 21:26:59.788523 29048 leader_election.cc:302] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9846ea932cb14dff98f20d67f1829791; no voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58
I20240502 21:26:59.788748 29247 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.790360 29247 raft_consensus.cc:2726] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:59.885026 29211 raft_consensus.cc:482] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.885211 29211 raft_consensus.cc:504] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.885808 29211 leader_election.cc:288] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:26:59.886243 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58" is_pre_election: true
I20240502 21:26:59.886241 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791" is_pre_election: true
I20240502 21:26:59.886482 29006 raft_consensus.cc:2454] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 0.
I20240502 21:26:59.886487 29083 raft_consensus.cc:2454] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 0.
I20240502 21:26:59.886837 29121 leader_election.cc:302] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:26:59.887118 29211 raft_consensus.cc:2781] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.887135 29237 raft_consensus.cc:482] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.887239 29211 raft_consensus.cc:482] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.887338 29211 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.887292 29237 raft_consensus.cc:504] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.887846 29237 leader_election.cc:288] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.888214 29152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a27d02080d420382add870508aaad4" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" is_pre_election: true
I20240502 21:26:59.888218 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a27d02080d420382add870508aaad4" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791" is_pre_election: true
I20240502 21:26:59.888464 29083 raft_consensus.cc:2454] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 0.
I20240502 21:26:59.888469 29152 raft_consensus.cc:2454] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 0.
I20240502 21:26:59.888806 28972 leader_election.cc:302] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:26:59.889037 29237 raft_consensus.cc:2781] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.889142 29237 raft_consensus.cc:482] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.889228 29237 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.890107 29211 raft_consensus.cc:504] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.890728 29211 leader_election.cc:288] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:26:59.891096 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:26:59.891278 29006 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.891265 29083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:26:59.891439 29083 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.891654 29237 raft_consensus.cc:504] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.892246 29237 leader_election.cc:288] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:26:59.892807 29082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a27d02080d420382add870508aaad4" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:26:59.893014 29082 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.893560 29006 raft_consensus.cc:2454] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 1.
I20240502 21:26:59.893889 29121 leader_election.cc:302] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:26:59.893929 29152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a27d02080d420382add870508aaad4" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:26:59.893993 29083 raft_consensus.cc:2454] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 1.
I20240502 21:26:59.894140 29152 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.894320 29211 raft_consensus.cc:2781] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.894629 29211 raft_consensus.cc:686] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:26:59.894985 29211 consensus_queue.cc:237] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.895334 29082 raft_consensus.cc:2454] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 1.
I20240502 21:26:59.895671 28972 leader_election.cc:302] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:26:59.895928 29237 raft_consensus.cc:2781] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.896051 29237 raft_consensus.cc:686] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:26:59.896334 29237 consensus_queue.cc:237] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:26:59.896783 29152 raft_consensus.cc:2454] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 1.
I20240502 21:26:59.898943 28903 catalog_manager.cc:5484] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 1 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:59.901844 28904 catalog_manager.cc:5484] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 1 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.994132 29237 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.023972 29247 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 21:27:00.048148 29237 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 21:27:00.090804 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:00.094458 29237 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20240502 21:27:00.107483 29007 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.108247 29209 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.135257 29152 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.136116 29154 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.157675 29247 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240502 21:27:00.158232 29247 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.162660 29209 raft_consensus.cc:482] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.163277 29209 raft_consensus.cc:504] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:00.220309 29209 leader_election.cc:288] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:00.220854 29006 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.221450 29256 consensus_queue.cc:1035] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.221787 29257 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.221844 29006 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
new_leader_uuid: "9846ea932cb14dff98f20d67f1829791"
from {username='slave'} at 127.0.0.1:52268
I20240502 21:27:00.222030 29256 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.222281 29006 raft_consensus.cc:595] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Received request to transfer leadership to TS 9846ea932cb14dff98f20d67f1829791
I20240502 21:27:00.222285 29257 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.222896 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 21:27:00.223156 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 21:27:00.216871 28972 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.16.66:46109, user_credentials={real_user=slave}} blocked reactor thread for 66883.4us
I20240502 21:27:00.225314 29154 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.226135 29154 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.229171 29154 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.230048 29007 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.230931 29295 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.231287 29297 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.231839 29295 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.232569 29295 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.234217 29295 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.240670 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f26f2c271544729391496f0fa81a58" is_pre_election: true
I20240502 21:27:00.240989 29007 raft_consensus.cc:2379] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9846ea932cb14dff98f20d67f1829791 in current term 1: Already voted for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in this term.
W20240502 21:27:00.241442 29048 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.16.67:41785, user_credentials={real_user=slave}} blocked reactor thread for 91847.8us
I20240502 21:27:00.241875 29211 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.243965 29082 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.245371 29157 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.245689 29153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" is_pre_election: true
I20240502 21:27:00.246347 29082 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.247014 29082 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.249907 29082 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.253011 29082 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.253763 29082 raft_consensus.cc:1261] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.254396 29082 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.262645 29247 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.262742 29256 consensus_queue.cc:1035] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.263286 29256 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.262679 29211 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 21:27:00.321805 29047 leader_election.cc:302] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9846ea932cb14dff98f20d67f1829791; no voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58
I20240502 21:27:00.327145 29295 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20240502 21:27:00.328689 29296 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
W20240502 21:27:00.329306 29121 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.16.65:33159, user_credentials={real_user=slave}} blocked reactor thread for 86533.6us
I20240502 21:27:00.329854 29298 raft_consensus.cc:2726] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:00.333474 29290 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20240502 21:27:00.410431 29279 mvcc.cc:205] Tried to move back new op lower bound from 7023350661682593792 to 7023350659840753664. Current Snapshot: MvccSnapshot[applied={T|T < 7023350661682593792}]
I20240502 21:27:00.446094 29237 raft_consensus.cc:979] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58: : Instructing follower 9846ea932cb14dff98f20d67f1829791 to start an election
I20240502 21:27:00.446300 29299 raft_consensus.cc:1067] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Signalling peer 9846ea932cb14dff98f20d67f1829791 to start an election
I20240502 21:27:00.450332 29077 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
from {username='slave'} at 127.0.0.1:51594
I20240502 21:27:00.450536 29077 raft_consensus.cc:482] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:00.450671 29077 raft_consensus.cc:3037] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:00.456154 29077 raft_consensus.cc:504] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:00.464501 29081 raft_consensus.cc:1226] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Rejecting Update request from peer 94f26f2c271544729391496f0fa81a58 for earlier term 1. Current term is 2. Ops: []
I20240502 21:27:00.464819 29077 leader_election.cc:288] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 2 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:00.467195 29150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:00.467396 29150 raft_consensus.cc:3037] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:27:00.469481 29305 log.cc:924] Time spent T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20240502 21:27:00.470329 29150 raft_consensus.cc:2454] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 2.
I20240502 21:27:00.471977 29048 leader_election.cc:302] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:00.472393 29247 raft_consensus.cc:2781] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:00.472599 29247 raft_consensus.cc:686] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Becoming Leader. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:00.472892 29247 consensus_queue.cc:237] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:00.476202 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:00.476450 29005 raft_consensus.cc:3032] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:00.476554 29005 raft_consensus.cc:729] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:00.476825 29005 consensus_queue.cc:260] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:00.477396 29005 raft_consensus.cc:3037] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:27:00.477684 29316 consensus_queue.cc:1159] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9846ea932cb14dff98f20d67f1829791" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 0 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 94f26f2c271544729391496f0fa81a58 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20240502 21:27:00.479699 28986 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52264: Illegal state: replica 94f26f2c271544729391496f0fa81a58 is not leader of this config: current role FOLLOWER
I20240502 21:27:00.480497 29005 raft_consensus.cc:2454] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 2.
W20240502 21:27:00.482422 28979 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52264: Illegal state: replica 94f26f2c271544729391496f0fa81a58 is not leader of this config: current role FOLLOWER
W20240502 21:27:00.482988 28987 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52264: Illegal state: replica 94f26f2c271544729391496f0fa81a58 is not leader of this config: current role FOLLOWER
I20240502 21:27:00.485255 28904 catalog_manager.cc:5484] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: term changed from 1 to 2, leader changed from 94f26f2c271544729391496f0fa81a58 (127.25.16.65) to 9846ea932cb14dff98f20d67f1829791 (127.25.16.66). New cstate: current_term: 2 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:00.511133 28987 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52264: Illegal state: replica 94f26f2c271544729391496f0fa81a58 is not leader of this config: current role FOLLOWER
I20240502 21:27:00.553177 29150 raft_consensus.cc:1226] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Rejecting Update request from peer 94f26f2c271544729391496f0fa81a58 for earlier term 1. Current term is 2. Ops: []
I20240502 21:27:00.588398 29155 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:00.605087 29007 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:00.616560 29209 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.011s
I20240502 21:27:00.616839 29247 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:00.675251 29289 mvcc.cc:205] Tried to move back new op lower bound from 7023350663579877376 to 7023350663059668992. Current Snapshot: MvccSnapshot[applied={T|T < 7023350663579877376}]
W20240502 21:27:00.679033 29048 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.16.67:41785, user_credentials={real_user=slave}} blocked reactor thread for 72905.1us
I20240502 21:27:01.206933 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:01.210106 29356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:01.210544 29357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:01.210955 29359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:01.211315 25665 server_base.cc:1034] running on GCE node
I20240502 21:27:01.211499 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:27:01.211545 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:27:01.211572 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685221211569 us; error 0 us; skew 500 ppm
I20240502 21:27:01.211679 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:01.212916 25665 webserver.cc:460] Webserver started at http://127.25.16.68:46011/ using document root <none> and password file <none>
I20240502 21:27:01.213107 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:01.213160 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:01.213254 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:01.214057 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-3-root/instance:
uuid: "c0c2fc14113d41c498fd6abcbbbd03c7"
format_stamp: "Formatted at 2024-05-02 21:27:01 on dist-test-slave-b524"
I20240502 21:27:01.217377 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:27:01.219295 29364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:01.219635 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 21:27:01.219735 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-3-root
uuid: "c0c2fc14113d41c498fd6abcbbbd03c7"
format_stamp: "Formatted at 2024-05-02 21:27:01 on dist-test-slave-b524"
I20240502 21:27:01.219841 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:01.233312 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:01.233868 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:01.234426 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:01.235296 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:01.235358 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:01.235426 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:01.235462 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:01.248694 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.68:45499
I20240502 21:27:01.248739 29426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.68:45499 every 8 connection(s)
I20240502 21:27:01.250339 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:01.253499 29431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:01.254683 29434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:01.255101 29432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:01.255550 25665 server_base.cc:1034] running on GCE node
I20240502 21:27:01.255776 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:27:01.255828 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:27:01.255860 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685221255856 us; error 0 us; skew 500 ppm
I20240502 21:27:01.255996 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:01.257545 25665 webserver.cc:460] Webserver started at http://127.25.16.69:44999/ using document root <none> and password file <none>
I20240502 21:27:01.257826 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:01.257910 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:01.258050 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:01.258816 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-4-root/instance:
uuid: "501feae33ef0484f956ca7a1e090ae01"
format_stamp: "Formatted at 2024-05-02 21:27:01 on dist-test-slave-b524"
I20240502 21:27:01.259845 29427 heartbeater.cc:347] Connected to a master server at 127.25.16.126:42491
I20240502 21:27:01.259972 29427 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:01.260412 29427 heartbeater.cc:511] Master 127.25.16.126:42491 requested a full tablet report, sending...
I20240502 21:27:01.261332 28904 ts_manager.cc:194] Registered new tserver with Master: c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499)
I20240502 21:27:01.261583 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 21:27:01.262398 28904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56908
I20240502 21:27:01.263438 29439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:01.263748 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 21:27:01.263844 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-4-root
uuid: "501feae33ef0484f956ca7a1e090ae01"
format_stamp: "Formatted at 2024-05-02 21:27:01 on dist-test-slave-b524"
I20240502 21:27:01.263947 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:01.272560 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:01.273139 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:01.273703 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:01.274578 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:01.274637 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:01.274708 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:01.274741 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:01.289525 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.69:35297
I20240502 21:27:01.289561 29502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.69:35297 every 8 connection(s)
I20240502 21:27:01.291208 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:01.294461 29506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:01.294844 29507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:01.301689 29509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:01.302217 25665 server_base.cc:1034] running on GCE node
I20240502 21:27:01.302477 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:27:01.302534 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:27:01.302577 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685221302573 us; error 0 us; skew 500 ppm
I20240502 21:27:01.302713 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:01.304467 25665 webserver.cc:460] Webserver started at http://127.25.16.70:39843/ using document root <none> and password file <none>
I20240502 21:27:01.304738 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:01.304857 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:01.305019 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:01.305444 29503 heartbeater.cc:347] Connected to a master server at 127.25.16.126:42491
I20240502 21:27:01.305578 29503 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:01.305845 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-5-root/instance:
uuid: "9c60776a34a54a09822291687d162a23"
format_stamp: "Formatted at 2024-05-02 21:27:01 on dist-test-slave-b524"
I20240502 21:27:01.306052 29503 heartbeater.cc:511] Master 127.25.16.126:42491 requested a full tablet report, sending...
I20240502 21:27:01.306968 28904 ts_manager.cc:194] Registered new tserver with Master: 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297)
I20240502 21:27:01.307705 28904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56910
I20240502 21:27:01.308748 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:27:01.310448 29514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:01.310801 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:01.310889 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-5-root
uuid: "9c60776a34a54a09822291687d162a23"
format_stamp: "Formatted at 2024-05-02 21:27:01 on dist-test-slave-b524"
I20240502 21:27:01.310986 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:01.335812 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:01.336442 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:01.337035 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:01.337925 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:01.337985 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:01.338063 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:01.338100 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:01.354029 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.70:35791
I20240502 21:27:01.354074 29577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.70:35791 every 8 connection(s)
I20240502 21:27:01.361474 29578 heartbeater.cc:347] Connected to a master server at 127.25.16.126:42491
I20240502 21:27:01.361621 29578 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:01.362063 29578 heartbeater.cc:511] Master 127.25.16.126:42491 requested a full tablet report, sending...
I20240502 21:27:01.362957 28904 ts_manager.cc:194] Registered new tserver with Master: 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791)
I20240502 21:27:01.363713 28904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56924
I20240502 21:27:02.223546 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:02.231881 29081 consensus_queue.cc:237] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:02.232678 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
new_leader_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:42850
I20240502 21:27:02.232878 29157 raft_consensus.cc:595] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Received request to transfer leadership to TS 94f26f2c271544729391496f0fa81a58
I20240502 21:27:02.233996 29003 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54a27d02080d420382add870508aaad4"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
new_leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:52268
I20240502 21:27:02.234195 29003 raft_consensus.cc:595] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Received request to transfer leadership to TS 5f1d764da7eb4f0c9fd4cabdca685d7b
I20240502 21:27:02.234637 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 21:27:02.234962 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:02.244695 29157 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20240502 21:27:02.245893 29003 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20240502 21:27:02.246161 29587 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240502 21:27:02.246824 29587 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240502 21:27:02.253237 29586 raft_consensus.cc:979] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: : Instructing follower 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:02.253453 29257 raft_consensus.cc:1067] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Signalling peer 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:02.254256 29003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:52250
I20240502 21:27:02.254464 29003 raft_consensus.cc:482] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:02.254562 29003 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:27:02.256428 29047 consensus_peers.cc:485] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 -> Peer 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791): Couldn't send request to peer 9c60776a34a54a09822291687d162a23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1de7fda467344968ab81f653b8501210. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:02.257229 29003 raft_consensus.cc:504] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:02.257901 29003 leader_election.cc:288] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 2 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:02.257757 29353 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:02.259155 29081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:02.259310 29081 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:02.258524 29150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:02.258337 29157 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:02.260447 29150 raft_consensus.cc:3032] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:02.260517 29150 raft_consensus.cc:729] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:02.260761 29150 consensus_queue.cc:260] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:02.261117 29150 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:02.262275 29081 raft_consensus.cc:2454] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 2.
I20240502 21:27:02.258618 29001 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:02.263604 29427 heartbeater.cc:503] Master 127.25.16.126:42491 was elected leader, sending a full tablet report...
I20240502 21:27:02.264055 28972 leader_election.cc:302] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:02.264412 29316 raft_consensus.cc:2781] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:02.264464 28904 catalog_manager.cc:5484] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index -1 to 24, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New cstate: current_term: 1 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:02.265347 29150 raft_consensus.cc:2454] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 2.
I20240502 21:27:02.267863 29316 raft_consensus.cc:686] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:02.268214 29316 consensus_queue.cc:237] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:02.273095 28903 catalog_manager.cc:5484] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 1 to 2, leader changed from 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 2 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:02.278036 28953 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
W20240502 21:27:02.280687 28953 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 54a27d02080d420382add870508aaad4, TS 5f1d764da7eb4f0c9fd4cabdca685d7b -> TS 501feae33ef0484f956ca7a1e090ae01 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
W20240502 21:27:02.280803 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 54a27d02080d420382add870508aaad4, TS 5f1d764da7eb4f0c9fd4cabdca685d7b -> TS 501feae33ef0484f956ca7a1e090ae01 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
W20240502 21:27:02.283016 28953 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 31c63a7053fa45b69eba6d7f7e78139b, TS 9846ea932cb14dff98f20d67f1829791 -> TS c0c2fc14113d41c498fd6abcbbbd03c7 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
W20240502 21:27:02.283135 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 31c63a7053fa45b69eba6d7f7e78139b, TS 9846ea932cb14dff98f20d67f1829791 -> TS c0c2fc14113d41c498fd6abcbbbd03c7 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
W20240502 21:27:02.285292 28953 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ab35657135e44445aee4aa6961f70c27, TS 94f26f2c271544729391496f0fa81a58 -> TS 501feae33ef0484f956ca7a1e090ae01 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
W20240502 21:27:02.285399 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ab35657135e44445aee4aa6961f70c27, TS 94f26f2c271544729391496f0fa81a58 -> TS 501feae33ef0484f956ca7a1e090ae01 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
W20240502 21:27:02.287513 28953 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet af41f02157c94f55b2bc5090f566592e, TS 5f1d764da7eb4f0c9fd4cabdca685d7b -> TS 9c60776a34a54a09822291687d162a23 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
W20240502 21:27:02.287638 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet af41f02157c94f55b2bc5090f566592e, TS 5f1d764da7eb4f0c9fd4cabdca685d7b -> TS 9c60776a34a54a09822291687d162a23 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
W20240502 21:27:02.289912 28953 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6d08f3643db34c8ba4d4391b630be366, TS 9846ea932cb14dff98f20d67f1829791 -> TS c0c2fc14113d41c498fd6abcbbbd03c7 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
W20240502 21:27:02.290016 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6d08f3643db34c8ba4d4391b630be366, TS 9846ea932cb14dff98f20d67f1829791 -> TS c0c2fc14113d41c498fd6abcbbbd03c7 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:02.309032 29503 heartbeater.cc:503] Master 127.25.16.126:42491 was elected leader, sending a full tablet report...
I20240502 21:27:02.365018 29578 heartbeater.cc:503] Master 127.25.16.126:42491 was elected leader, sending a full tablet report...
I20240502 21:27:02.414404 29594 raft_consensus.cc:979] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58: : Instructing follower 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:02.414680 29316 raft_consensus.cc:1067] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Signalling peer 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:02.415298 29157 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54a27d02080d420382add870508aaad4"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:34468
I20240502 21:27:02.415514 29157 raft_consensus.cc:482] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:02.415639 29157 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:02.418664 29157 raft_consensus.cc:504] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:02.419684 29157 leader_election.cc:288] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 2 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:02.419750 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a27d02080d420382add870508aaad4" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:02.419975 29001 raft_consensus.cc:3032] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:02.420071 29001 raft_consensus.cc:729] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:02.419926 29076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a27d02080d420382add870508aaad4" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:02.420367 29001 consensus_queue.cc:260] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:02.420521 29076 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:02.420734 29001 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:02.422737 29076 raft_consensus.cc:2454] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 2.
I20240502 21:27:02.423115 29122 leader_election.cc:302] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:02.423341 29586 raft_consensus.cc:2781] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:02.423416 29001 raft_consensus.cc:2454] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 2.
I20240502 21:27:02.423506 29586 raft_consensus.cc:686] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:02.423848 29586 consensus_queue.cc:237] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:02.427242 28902 catalog_manager.cc:5484] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 1 to 2, leader changed from 94f26f2c271544729391496f0fa81a58 (127.25.16.65) to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 2 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:02.713300 29076 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:02.713806 29316 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:02.718822 29157 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:02.719583 29316 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:02.759245 29607 ts_tablet_manager.cc:902] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: Initiating tablet copy from peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:02.760752 29607 tablet_copy_client.cc:308] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.66:46109
I20240502 21:27:02.767608 29093 tablet_copy_service.cc:140] P 9846ea932cb14dff98f20d67f1829791: Received BeginTabletCopySession request for tablet 1de7fda467344968ab81f653b8501210 from peer 9c60776a34a54a09822291687d162a23 ({username='slave'} at 127.0.0.1:60728)
I20240502 21:27:02.767843 29093 tablet_copy_service.cc:161] P 9846ea932cb14dff98f20d67f1829791: Beginning new tablet copy session on tablet 1de7fda467344968ab81f653b8501210 from peer 9c60776a34a54a09822291687d162a23 at {username='slave'} at 127.0.0.1:60728: session id = 9c60776a34a54a09822291687d162a23-1de7fda467344968ab81f653b8501210
I20240502 21:27:02.770576 29093 tablet_copy_source_session.cc:205] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:02.771934 29607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de7fda467344968ab81f653b8501210. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.777962 29607 tablet_copy_client.cc:783] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:02.778285 29607 tablet_copy_client.cc:647] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:02.779637 29611 tablet_copy_client.cc:1002] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:02.850762 29081 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:02.851364 29257 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:02.855945 29001 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:02.856529 29615 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:04.235337 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:04.238798 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54a27d02080d420382add870508aaad4"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
new_leader_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:42850
I20240502 21:27:04.239018 29157 raft_consensus.cc:595] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Received request to transfer leadership to TS 94f26f2c271544729391496f0fa81a58
I20240502 21:27:04.240001 29001 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
new_leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:52268
I20240502 21:27:04.240191 29001 raft_consensus.cc:595] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Received request to transfer leadership to TS 5f1d764da7eb4f0c9fd4cabdca685d7b
I20240502 21:27:04.240674 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 21:27:04.240875 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:04.301409 29620 raft_consensus.cc:979] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b: : Instructing follower 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:04.301563 29620 raft_consensus.cc:1067] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Signalling peer 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:04.302119 29001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54a27d02080d420382add870508aaad4"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:52250
I20240502 21:27:04.302291 29001 raft_consensus.cc:482] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:04.302400 29001 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:04.304725 29001 raft_consensus.cc:504] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:04.305371 29001 leader_election.cc:288] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 3 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:04.305701 29081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a27d02080d420382add870508aaad4" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:04.305795 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a27d02080d420382add870508aaad4" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:04.306013 29081 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:04.306044 29157 raft_consensus.cc:3032] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:04.306114 29157 raft_consensus.cc:729] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:04.306365 29157 consensus_queue.cc:260] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:04.306619 29157 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:04.307916 29081 raft_consensus.cc:2454] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 3.
I20240502 21:27:04.308315 29157 raft_consensus.cc:2454] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 3.
I20240502 21:27:04.308437 28972 leader_election.cc:302] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:04.308704 29617 raft_consensus.cc:2781] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:04.309111 29617 raft_consensus.cc:686] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:04.309374 29617 consensus_queue.cc:237] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:04.311982 28902 catalog_manager.cc:5484] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 2 to 3, leader changed from 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 3 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:04.780050 29607 tablet_copy_client.cc:523] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:04.783425 29607 tablet_bootstrap.cc:492] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: Bootstrap starting.
I20240502 21:27:04.819166 29607 tablet_bootstrap.cc:492] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=91 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:04.819583 29607 tablet_bootstrap.cc:492] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: Bootstrap complete.
I20240502 21:27:04.819844 29607 ts_tablet_manager.cc:1367] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: Time spent bootstrapping tablet: real 0.036s user 0.026s sys 0.009s
I20240502 21:27:04.820624 29607 raft_consensus.cc:348] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:04.820870 29607 raft_consensus.cc:729] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9c60776a34a54a09822291687d162a23, State: Initialized, Role: LEARNER
I20240502 21:27:04.821060 29607 consensus_queue.cc:260] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:04.821846 29607 ts_tablet_manager.cc:1398] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 21:27:04.822499 29093 tablet_copy_service.cc:342] P 9846ea932cb14dff98f20d67f1829791: Request end of tablet copy session 9c60776a34a54a09822291687d162a23-1de7fda467344968ab81f653b8501210 received from {username='slave'} at 127.0.0.1:60728
I20240502 21:27:04.822610 29093 tablet_copy_service.cc:433] P 9846ea932cb14dff98f20d67f1829791: ending tablet copy session 9c60776a34a54a09822291687d162a23-1de7fda467344968ab81f653b8501210 on tablet 1de7fda467344968ab81f653b8501210 with peer 9c60776a34a54a09822291687d162a23
I20240502 21:27:04.828994 29630 raft_consensus.cc:979] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58: : Instructing follower 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:04.829174 29617 raft_consensus.cc:1067] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Signalling peer 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:04.829764 29157 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:34468
I20240502 21:27:04.829947 29157 raft_consensus.cc:482] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:04.830034 29157 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.832394 29157 raft_consensus.cc:504] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:04.833047 29157 leader_election.cc:288] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 2 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:04.833281 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:04.833438 29001 raft_consensus.cc:3032] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:04.833503 29001 raft_consensus.cc:729] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:04.833729 29001 consensus_queue.cc:260] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:04.833994 29001 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.835913 29001 raft_consensus.cc:2454] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 2.
I20240502 21:27:04.836359 29121 leader_election.cc:302] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:27:04.836541 29076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:04.836678 29620 raft_consensus.cc:2781] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:04.836720 29076 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.837052 29620 raft_consensus.cc:686] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:04.837363 29620 consensus_queue.cc:237] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:04.839486 29076 raft_consensus.cc:2454] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 2.
I20240502 21:27:04.840205 28902 catalog_manager.cc:5484] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 1 to 2, leader changed from 94f26f2c271544729391496f0fa81a58 (127.25.16.65) to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 2 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:04.845330 29157 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 21:27:04.845880 29617 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:04.858170 29076 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 21:27:04.859838 29630 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:05.208845 29552 raft_consensus.cc:1203] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.23->[1.24-1.24] Dedup: 1.24->[]
I20240502 21:27:05.337118 29001 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:05.337672 29620 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:05.342658 29081 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:05.343171 29620 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:05.700410 29627 raft_consensus.cc:1050] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791: attempting to promote NON_VOTER 9c60776a34a54a09822291687d162a23 to VOTER
I20240502 21:27:05.701277 29627 consensus_queue.cc:237] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:05.703471 29552 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 1 LEARNER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 21:27:05.703488 29001 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 21:27:05.703451 29150 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 21:27:05.703986 29649 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 21:27:05.704243 29648 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 21:27:05.704773 29648 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 21:27:05.718089 29651 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:05.718746 29552 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:05.719326 29001 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:05.723651 29150 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:05.724016 28903 catalog_manager.cc:5484] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index 24 to 25, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:05.732394 29076 consensus_queue.cc:237] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:05.733919 29150 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 21:27:05.734413 29552 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 21:27:05.735896 29649 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:05.736187 29651 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:05.738328 29627 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:05.738902 29552 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:05.739997 29150 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:05.743609 28903 catalog_manager.cc:5484] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 reported cstate change: config changed from index 25 to 26, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New cstate: current_term: 1 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:05.745815 28890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1de7fda467344968ab81f653b8501210 with cas_config_opid_index 25: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:05.748394 28994 tablet_service.cc:1514] Processing DeleteTablet for tablet 1de7fda467344968ab81f653b8501210 with delete_type TABLET_DATA_TOMBSTONED (TS 94f26f2c271544729391496f0fa81a58 not found in new config with opid_index 26) from {username='slave'} at 127.0.0.1:52236
I20240502 21:27:05.749022 29659 tablet_replica.cc:330] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58: stopping tablet replica
I20240502 21:27:05.749204 29659 raft_consensus.cc:2229] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.749486 29659 raft_consensus.cc:2258] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.750299 29659 ts_tablet_manager.cc:1875] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:05.754777 29659 ts_tablet_manager.cc:1888] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.25
I20240502 21:27:05.754882 29659 log.cc:1196] T 1de7fda467344968ab81f653b8501210 P 94f26f2c271544729391496f0fa81a58: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-0-root/wals/1de7fda467344968ab81f653b8501210
I20240502 21:27:05.755468 28889 catalog_manager.cc:4830] TS 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): tablet 1de7fda467344968ab81f653b8501210 (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:06.241271 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:06.244235 29001 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5010a09a5243452dad7cd9b90535a229"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
new_leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:52268
I20240502 21:27:06.244498 29001 raft_consensus.cc:595] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Received request to transfer leadership to TS 5f1d764da7eb4f0c9fd4cabdca685d7b
I20240502 21:27:06.245653 29150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
new_leader_uuid: "9846ea932cb14dff98f20d67f1829791"
from {username='slave'} at 127.0.0.1:42850
I20240502 21:27:06.245841 29150 raft_consensus.cc:595] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Received request to transfer leadership to TS 9846ea932cb14dff98f20d67f1829791
I20240502 21:27:06.246294 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 21:27:06.246472 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:06.474095 29642 raft_consensus.cc:979] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b: : Instructing follower 9846ea932cb14dff98f20d67f1829791 to start an election
I20240502 21:27:06.474233 29642 raft_consensus.cc:1067] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Signalling peer 9846ea932cb14dff98f20d67f1829791 to start an election
I20240502 21:27:06.474812 29081 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
from {username='slave'} at 127.0.0.1:51598
I20240502 21:27:06.475006 29081 raft_consensus.cc:482] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:06.475093 29081 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:06.477038 29081 raft_consensus.cc:504] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:06.477579 29081 leader_election.cc:288] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 2 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:06.477999 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:06.478173 29001 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:06.478124 29150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:06.478305 29150 raft_consensus.cc:3032] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:06.478363 29150 raft_consensus.cc:729] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:06.478591 29150 consensus_queue.cc:260] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:06.478886 29150 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:06.480541 29001 raft_consensus.cc:2454] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 2.
I20240502 21:27:06.480743 29150 raft_consensus.cc:2454] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 2.
I20240502 21:27:06.480968 29047 leader_election.cc:302] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:06.481220 29627 raft_consensus.cc:2781] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:06.481529 29627 raft_consensus.cc:686] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Becoming Leader. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:06.481786 29627 consensus_queue.cc:237] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:06.485141 28902 catalog_manager.cc:5484] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: term changed from 1 to 2, leader changed from 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) to 9846ea932cb14dff98f20d67f1829791 (127.25.16.66). New cstate: current_term: 2 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:06.686944 29630 raft_consensus.cc:979] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58: : Instructing follower 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:06.687117 29666 raft_consensus.cc:1067] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Signalling peer 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:06.687803 29150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5010a09a5243452dad7cd9b90535a229"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:34468
I20240502 21:27:06.687987 29150 raft_consensus.cc:482] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:06.688078 29150 raft_consensus.cc:3037] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:06.690039 29150 raft_consensus.cc:504] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } }
I20240502 21:27:06.690656 29150 leader_election.cc:288] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 2 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:06.691112 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5010a09a5243452dad7cd9b90535a229" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:06.691112 29081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5010a09a5243452dad7cd9b90535a229" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:06.691340 29001 raft_consensus.cc:3032] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:06.691380 29081 raft_consensus.cc:3037] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:06.691404 29001 raft_consensus.cc:729] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:06.691655 29001 consensus_queue.cc:260] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } }
I20240502 21:27:06.691949 29001 raft_consensus.cc:3037] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:06.693491 29081 raft_consensus.cc:2454] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 2.
I20240502 21:27:06.693910 29122 leader_election.cc:302] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:06.693919 29001 raft_consensus.cc:2454] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 2.
I20240502 21:27:06.694178 29642 raft_consensus.cc:2781] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:06.694308 29642 raft_consensus.cc:686] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:06.694542 29642 consensus_queue.cc:237] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } }
I20240502 21:27:06.697054 28902 catalog_manager.cc:5484] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 1 to 2, leader changed from 94f26f2c271544729391496f0fa81a58 (127.25.16.65) to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 2 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:06.762185 29150 consensus_queue.cc:237] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:06.764957 29001 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:06.765630 29671 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:06.765777 29081 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:06.766281 29642 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:06.769006 29671 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:06.769662 29001 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:06.770488 29081 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
W20240502 21:27:06.773655 29122 consensus_peers.cc:485] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b -> Peer 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297): Couldn't send request to peer 501feae33ef0484f956ca7a1e090ae01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af41f02157c94f55b2bc5090f566592e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:06.775521 28902 catalog_manager.cc:5484] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index -1 to 27, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New cstate: current_term: 2 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:06.776796 29081 consensus_queue.cc:237] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:06.780218 29001 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 21:27:06.780874 29627 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 21:27:06.781414 29150 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 21:27:06.782322 29627 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 21:27:06.795068 29627 raft_consensus.cc:2932] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
W20240502 21:27:06.800338 29046 consensus_peers.cc:485] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 -> Peer c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499): Couldn't send request to peer c0c2fc14113d41c498fd6abcbbbd03c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee4eba5acb8d4af986a00f937de0270e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:06.801427 29157 raft_consensus.cc:2932] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:06.803973 28903 catalog_manager.cc:5484] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index -1 to 25, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New cstate: current_term: 1 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:06.804268 29003 raft_consensus.cc:2932] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:06.809219 29001 consensus_queue.cc:237] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:06.813078 29081 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 21:27:06.813086 29157 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 21:27:06.813647 29630 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:06.813971 29666 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:06.817341 29630 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:06.818050 29081 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:06.822873 28904 catalog_manager.cc:5484] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index -1 to 28, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New cstate: current_term: 3 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:06.824956 29001 consensus_queue.cc:237] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
W20240502 21:27:06.829320 28971 consensus_peers.cc:485] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 -> Peer 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791): Couldn't send request to peer 9c60776a34a54a09822291687d162a23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54a27d02080d420382add870508aaad4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:06.833729 29081 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:06.833729 29150 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:06.834571 29683 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:06.834357 29157 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:06.835069 29666 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:06.837759 29666 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:06.841514 29157 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:06.842425 29081 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
W20240502 21:27:06.843104 28972 consensus_peers.cc:485] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 -> Peer 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297): Couldn't send request to peer 501feae33ef0484f956ca7a1e090ae01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31c63a7053fa45b69eba6d7f7e78139b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:06.846452 28903 catalog_manager.cc:5484] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index -1 to 27, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New cstate: current_term: 2 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:06.847831 29081 consensus_queue.cc:237] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:06.850265 29001 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 21:27:06.850307 29157 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 21:27:06.850709 29661 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
W20240502 21:27:06.850977 29046 consensus_peers.cc:485] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 -> Peer c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499): Couldn't send request to peer c0c2fc14113d41c498fd6abcbbbd03c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccd9ffc92c5c47d088c530e919f2ff77. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:06.850997 29627 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:06.853773 29627 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:06.854413 29157 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:06.855175 29003 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:06.858909 29081 consensus_queue.cc:237] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:06.860006 28903 catalog_manager.cc:5484] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index -1 to 26, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New cstate: current_term: 1 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:06.863147 29003 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:06.863651 29627 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:06.867035 29157 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:06.866588 29661 raft_consensus.cc:2932] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:06.867206 29003 raft_consensus.cc:2932] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:06.867537 29677 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:06.869745 29150 raft_consensus.cc:2932] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
W20240502 21:27:06.871028 29048 consensus_peers.cc:485] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 -> Peer 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297): Couldn't send request to peer 501feae33ef0484f956ca7a1e090ae01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab35657135e44445aee4aa6961f70c27. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:06.873064 28904 catalog_manager.cc:5484] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index -1 to 26, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New cstate: current_term: 2 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:06.888695 29003 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:06.889212 29677 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:06.893821 29150 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:06.894348 29661 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:07.195756 29003 raft_consensus.cc:1261] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:07.196261 29671 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:07.200202 29081 raft_consensus.cc:1261] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:07.201599 29660 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:07.209754 29704 ts_tablet_manager.cc:902] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7: Initiating tablet copy from peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:07.217607 29704 tablet_copy_client.cc:308] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.66:46109
I20240502 21:27:07.224033 29093 tablet_copy_service.cc:140] P 9846ea932cb14dff98f20d67f1829791: Received BeginTabletCopySession request for tablet ee4eba5acb8d4af986a00f937de0270e from peer c0c2fc14113d41c498fd6abcbbbd03c7 ({username='slave'} at 127.0.0.1:60738)
I20240502 21:27:07.224198 29093 tablet_copy_service.cc:161] P 9846ea932cb14dff98f20d67f1829791: Beginning new tablet copy session on tablet ee4eba5acb8d4af986a00f937de0270e from peer c0c2fc14113d41c498fd6abcbbbd03c7 at {username='slave'} at 127.0.0.1:60738: session id = c0c2fc14113d41c498fd6abcbbbd03c7-ee4eba5acb8d4af986a00f937de0270e
I20240502 21:27:07.226128 29093 tablet_copy_source_session.cc:205] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:07.227206 29704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee4eba5acb8d4af986a00f937de0270e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.232659 29704 tablet_copy_client.cc:783] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:07.232897 29704 tablet_copy_client.cc:647] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:07.234367 29708 tablet_copy_client.cc:1002] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:07.336562 29712 ts_tablet_manager.cc:902] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: Initiating tablet copy from peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:07.338521 29712 tablet_copy_client.cc:308] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.67:41785
I20240502 21:27:07.346786 29167 tablet_copy_service.cc:140] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Received BeginTabletCopySession request for tablet af41f02157c94f55b2bc5090f566592e from peer 501feae33ef0484f956ca7a1e090ae01 ({username='slave'} at 127.0.0.1:42866)
I20240502 21:27:07.346994 29167 tablet_copy_service.cc:161] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Beginning new tablet copy session on tablet af41f02157c94f55b2bc5090f566592e from peer 501feae33ef0484f956ca7a1e090ae01 at {username='slave'} at 127.0.0.1:42866: session id = 501feae33ef0484f956ca7a1e090ae01-af41f02157c94f55b2bc5090f566592e
I20240502 21:27:07.347368 29711 ts_tablet_manager.cc:902] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: Initiating tablet copy from peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159)
I20240502 21:27:07.350447 29167 tablet_copy_source_session.cc:205] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:07.351748 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af41f02157c94f55b2bc5090f566592e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.354801 29711 tablet_copy_client.cc:308] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.65:33159
I20240502 21:27:07.358075 29712 tablet_copy_client.cc:783] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:07.358352 29712 tablet_copy_client.cc:647] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:07.361256 29716 tablet_copy_client.cc:1002] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:07.362941 29017 tablet_copy_service.cc:140] P 94f26f2c271544729391496f0fa81a58: Received BeginTabletCopySession request for tablet 54a27d02080d420382add870508aaad4 from peer 9c60776a34a54a09822291687d162a23 ({username='slave'} at 127.0.0.1:48054)
I20240502 21:27:07.363085 29017 tablet_copy_service.cc:161] P 94f26f2c271544729391496f0fa81a58: Beginning new tablet copy session on tablet 54a27d02080d420382add870508aaad4 from peer 9c60776a34a54a09822291687d162a23 at {username='slave'} at 127.0.0.1:48054: session id = 9c60776a34a54a09822291687d162a23-54a27d02080d420382add870508aaad4
I20240502 21:27:07.365195 29017 tablet_copy_source_session.cc:205] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:07.366422 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a27d02080d420382add870508aaad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.372843 29711 tablet_copy_client.cc:783] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:07.373119 29711 tablet_copy_client.cc:647] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:07.374465 29722 tablet_copy_client.cc:1002] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:07.417541 29725 ts_tablet_manager.cc:902] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01: Initiating tablet copy from peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:07.418921 29725 tablet_copy_client.cc:308] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.66:46109
I20240502 21:27:07.424374 29093 tablet_copy_service.cc:140] P 9846ea932cb14dff98f20d67f1829791: Received BeginTabletCopySession request for tablet ab35657135e44445aee4aa6961f70c27 from peer 501feae33ef0484f956ca7a1e090ae01 ({username='slave'} at 127.0.0.1:60746)
I20240502 21:27:07.424533 29093 tablet_copy_service.cc:161] P 9846ea932cb14dff98f20d67f1829791: Beginning new tablet copy session on tablet ab35657135e44445aee4aa6961f70c27 from peer 501feae33ef0484f956ca7a1e090ae01 at {username='slave'} at 127.0.0.1:60746: session id = 501feae33ef0484f956ca7a1e090ae01-ab35657135e44445aee4aa6961f70c27
I20240502 21:27:07.427446 29093 tablet_copy_source_session.cc:205] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:07.428593 29725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab35657135e44445aee4aa6961f70c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.432466 29730 ts_tablet_manager.cc:902] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01: Initiating tablet copy from peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159)
I20240502 21:27:07.434306 29730 tablet_copy_client.cc:308] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.65:33159
I20240502 21:27:07.436419 29725 tablet_copy_client.cc:783] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:07.436650 29725 tablet_copy_client.cc:647] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:07.438318 29729 tablet_copy_client.cc:1002] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:07.441179 29017 tablet_copy_service.cc:140] P 94f26f2c271544729391496f0fa81a58: Received BeginTabletCopySession request for tablet 31c63a7053fa45b69eba6d7f7e78139b from peer 501feae33ef0484f956ca7a1e090ae01 ({username='slave'} at 127.0.0.1:48068)
I20240502 21:27:07.441298 29017 tablet_copy_service.cc:161] P 94f26f2c271544729391496f0fa81a58: Beginning new tablet copy session on tablet 31c63a7053fa45b69eba6d7f7e78139b from peer 501feae33ef0484f956ca7a1e090ae01 at {username='slave'} at 127.0.0.1:48068: session id = 501feae33ef0484f956ca7a1e090ae01-31c63a7053fa45b69eba6d7f7e78139b
I20240502 21:27:07.443708 29017 tablet_copy_source_session.cc:205] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:07.444842 29730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c63a7053fa45b69eba6d7f7e78139b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.449891 29735 ts_tablet_manager.cc:902] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: Initiating tablet copy from peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:07.451370 29730 tablet_copy_client.cc:783] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:07.451592 29730 tablet_copy_client.cc:647] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:07.453269 29734 tablet_copy_client.cc:1002] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:07.457185 29735 tablet_copy_client.cc:308] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.66:46109
I20240502 21:27:07.457820 29093 tablet_copy_service.cc:140] P 9846ea932cb14dff98f20d67f1829791: Received BeginTabletCopySession request for tablet ccd9ffc92c5c47d088c530e919f2ff77 from peer c0c2fc14113d41c498fd6abcbbbd03c7 ({username='slave'} at 127.0.0.1:60738)
I20240502 21:27:07.457968 29093 tablet_copy_service.cc:161] P 9846ea932cb14dff98f20d67f1829791: Beginning new tablet copy session on tablet ccd9ffc92c5c47d088c530e919f2ff77 from peer c0c2fc14113d41c498fd6abcbbbd03c7 at {username='slave'} at 127.0.0.1:60738: session id = c0c2fc14113d41c498fd6abcbbbd03c7-ccd9ffc92c5c47d088c530e919f2ff77
I20240502 21:27:07.460520 29093 tablet_copy_source_session.cc:205] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:07.461460 29735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd9ffc92c5c47d088c530e919f2ff77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.466557 29735 tablet_copy_client.cc:783] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:07.466791 29735 tablet_copy_client.cc:647] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:07.468178 29739 tablet_copy_client.cc:1002] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:08.246786 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:08.256088 29076 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1de7fda467344968ab81f653b8501210"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
mode: GRACEFUL
new_leader_uuid: "9c60776a34a54a09822291687d162a23"
from {username='slave'} at 127.0.0.1:60752
I20240502 21:27:08.256341 29076 raft_consensus.cc:595] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Received request to transfer leadership to TS 9c60776a34a54a09822291687d162a23
I20240502 21:27:08.257685 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
new_leader_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:42850
I20240502 21:27:08.257859 29157 raft_consensus.cc:595] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Received request to transfer leadership to TS 94f26f2c271544729391496f0fa81a58
I20240502 21:27:08.258436 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 21:27:08.258667 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:08.602607 29743 raft_consensus.cc:979] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b: : Instructing follower 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:08.602763 29743 raft_consensus.cc:1067] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Signalling peer 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:08.603286 29003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:52250
I20240502 21:27:08.603477 29003 raft_consensus.cc:482] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:08.603583 29003 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:08.605990 29003 raft_consensus.cc:504] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:08.606570 29003 leader_election.cc:288] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 3 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:08.606987 29081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:08.606986 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:08.607196 29081 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:08.607244 29157 raft_consensus.cc:3032] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:08.607317 29157 raft_consensus.cc:729] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:08.607555 29157 consensus_queue.cc:260] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:08.607959 29157 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:08.610646 29157 raft_consensus.cc:2454] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 3.
I20240502 21:27:08.610766 29081 raft_consensus.cc:2454] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 3.
I20240502 21:27:08.611040 28972 leader_election.cc:302] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:27:08.611255 29741 raft_consensus.cc:2781] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:08.611389 29741 raft_consensus.cc:686] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:08.611625 29741 consensus_queue.cc:237] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:08.615100 28902 catalog_manager.cc:5484] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 2 to 3, leader changed from 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 3 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:08.786372 29677 raft_consensus.cc:979] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791: : Instructing follower 9c60776a34a54a09822291687d162a23 to start an election
I20240502 21:27:08.786526 29677 raft_consensus.cc:1067] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Signalling peer 9c60776a34a54a09822291687d162a23 to start an election
I20240502 21:27:08.787135 29552 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1de7fda467344968ab81f653b8501210"
dest_uuid: "9c60776a34a54a09822291687d162a23"
from {username='slave'} at 127.0.0.1:56366
I20240502 21:27:08.787338 29552 raft_consensus.cc:482] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:08.787431 29552 raft_consensus.cc:3037] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:08.789873 29552 raft_consensus.cc:504] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:08.790681 29552 leader_election.cc:288] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 2 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:08.790935 29076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de7fda467344968ab81f653b8501210" candidate_uuid: "9c60776a34a54a09822291687d162a23" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:08.791599 29076 raft_consensus.cc:3032] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:08.791684 29076 raft_consensus.cc:729] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:08.791922 29076 consensus_queue.cc:260] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:08.792232 29076 raft_consensus.cc:3037] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:08.795251 29076 raft_consensus.cc:2454] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c60776a34a54a09822291687d162a23 in term 2.
I20240502 21:27:08.795652 29517 leader_election.cc:302] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9846ea932cb14dff98f20d67f1829791, 9c60776a34a54a09822291687d162a23; no voters:
I20240502 21:27:08.796262 29753 raft_consensus.cc:2781] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:08.796743 29753 raft_consensus.cc:686] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 2 LEADER]: Becoming Leader. State: Replica: 9c60776a34a54a09822291687d162a23, State: Running, Role: LEADER
I20240502 21:27:08.797504 29753 consensus_queue.cc:237] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:08.799001 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de7fda467344968ab81f653b8501210" candidate_uuid: "9c60776a34a54a09822291687d162a23" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:08.799167 29157 raft_consensus.cc:3037] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:08.801828 29157 raft_consensus.cc:2454] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c60776a34a54a09822291687d162a23 in term 2.
I20240502 21:27:08.801868 28902 catalog_manager.cc:5484] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 reported cstate change: term changed from 1 to 2, leader changed from 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) to 9c60776a34a54a09822291687d162a23 (127.25.16.70). New cstate: current_term: 2 leader_uuid: "9c60776a34a54a09822291687d162a23" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 21:27:08.999486 28972 consensus_peers.cc:485] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 -> Peer 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297): Couldn't send request to peer 501feae33ef0484f956ca7a1e090ae01. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:09.144069 29157 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 21:27:09.144640 29741 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:09.149406 29076 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 21:27:09.150856 29757 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:09.188843 29076 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 9c60776a34a54a09822291687d162a23: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:09.189378 29753 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:09.193571 29157 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 9c60776a34a54a09822291687d162a23: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:09.194047 29753 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:09.234752 29704 tablet_copy_client.cc:523] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:09.238157 29704 tablet_bootstrap.cc:492] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap starting.
I20240502 21:27:09.275003 29704 tablet_bootstrap.cc:492] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:09.275439 29704 tablet_bootstrap.cc:492] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap complete.
I20240502 21:27:09.275734 29704 ts_tablet_manager.cc:1367] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent bootstrapping tablet: real 0.038s user 0.027s sys 0.008s
I20240502 21:27:09.276412 29704 raft_consensus.cc:348] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:09.276628 29704 raft_consensus.cc:729] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c0c2fc14113d41c498fd6abcbbbd03c7, State: Initialized, Role: LEARNER
I20240502 21:27:09.276835 29704 consensus_queue.cc:260] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:09.277633 29704 ts_tablet_manager.cc:1398] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 21:27:09.278206 29093 tablet_copy_service.cc:342] P 9846ea932cb14dff98f20d67f1829791: Request end of tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-ee4eba5acb8d4af986a00f937de0270e received from {username='slave'} at 127.0.0.1:60738
I20240502 21:27:09.278324 29093 tablet_copy_service.cc:433] P 9846ea932cb14dff98f20d67f1829791: ending tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-ee4eba5acb8d4af986a00f937de0270e on tablet ee4eba5acb8d4af986a00f937de0270e with peer c0c2fc14113d41c498fd6abcbbbd03c7
I20240502 21:27:09.361639 29712 tablet_copy_client.cc:523] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:09.365000 29712 tablet_bootstrap.cc:492] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: Bootstrap starting.
I20240502 21:27:09.374799 29711 tablet_copy_client.cc:523] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:09.378191 29711 tablet_bootstrap.cc:492] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: Bootstrap starting.
I20240502 21:27:09.402374 29712 tablet_bootstrap.cc:492] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=94 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:09.402817 29712 tablet_bootstrap.cc:492] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: Bootstrap complete.
I20240502 21:27:09.403095 29712 ts_tablet_manager.cc:1367] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: Time spent bootstrapping tablet: real 0.038s user 0.029s sys 0.006s
I20240502 21:27:09.403829 29712 raft_consensus.cc:348] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:09.404083 29712 raft_consensus.cc:729] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 501feae33ef0484f956ca7a1e090ae01, State: Initialized, Role: LEARNER
I20240502 21:27:09.404366 29712 consensus_queue.cc:260] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:09.405912 29712 ts_tablet_manager.cc:1398] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: Time spent starting tablet: real 0.003s user 0.001s sys 0.004s
I20240502 21:27:09.406538 29167 tablet_copy_service.cc:342] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Request end of tablet copy session 501feae33ef0484f956ca7a1e090ae01-af41f02157c94f55b2bc5090f566592e received from {username='slave'} at 127.0.0.1:42866
I20240502 21:27:09.406632 29167 tablet_copy_service.cc:433] P 5f1d764da7eb4f0c9fd4cabdca685d7b: ending tablet copy session 501feae33ef0484f956ca7a1e090ae01-af41f02157c94f55b2bc5090f566592e on tablet af41f02157c94f55b2bc5090f566592e with peer 501feae33ef0484f956ca7a1e090ae01
I20240502 21:27:09.411257 29711 tablet_bootstrap.cc:492] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=85 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:09.411675 29711 tablet_bootstrap.cc:492] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: Bootstrap complete.
I20240502 21:27:09.411911 29711 ts_tablet_manager.cc:1367] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: Time spent bootstrapping tablet: real 0.034s user 0.034s sys 0.000s
I20240502 21:27:09.412459 29711 raft_consensus.cc:348] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:09.412633 29711 raft_consensus.cc:729] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 9c60776a34a54a09822291687d162a23, State: Initialized, Role: LEARNER
I20240502 21:27:09.412804 29711 consensus_queue.cc:260] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:09.413228 29711 ts_tablet_manager.cc:1398] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:09.413731 29017 tablet_copy_service.cc:342] P 94f26f2c271544729391496f0fa81a58: Request end of tablet copy session 9c60776a34a54a09822291687d162a23-54a27d02080d420382add870508aaad4 received from {username='slave'} at 127.0.0.1:48054
I20240502 21:27:09.413815 29017 tablet_copy_service.cc:433] P 94f26f2c271544729391496f0fa81a58: ending tablet copy session 9c60776a34a54a09822291687d162a23-54a27d02080d420382add870508aaad4 on tablet 54a27d02080d420382add870508aaad4 with peer 9c60776a34a54a09822291687d162a23
I20240502 21:27:09.438890 29725 tablet_copy_client.cc:523] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:09.442493 29725 tablet_bootstrap.cc:492] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01: Bootstrap starting.
I20240502 21:27:09.445314 29552 raft_consensus.cc:1203] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.27->[3.28-3.28] Dedup: 3.28->[]
I20240502 21:27:09.453605 29730 tablet_copy_client.cc:523] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:09.457006 29730 tablet_bootstrap.cc:492] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01: Bootstrap starting.
I20240502 21:27:09.468669 29735 tablet_copy_client.cc:523] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:09.472371 29735 tablet_bootstrap.cc:492] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap starting.
I20240502 21:27:09.481781 29725 tablet_bootstrap.cc:492] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:09.482226 29725 tablet_bootstrap.cc:492] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01: Bootstrap complete.
I20240502 21:27:09.482515 29725 ts_tablet_manager.cc:1367] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01: Time spent bootstrapping tablet: real 0.040s user 0.033s sys 0.007s
I20240502 21:27:09.483309 29725 raft_consensus.cc:348] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:09.483561 29725 raft_consensus.cc:729] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 501feae33ef0484f956ca7a1e090ae01, State: Initialized, Role: LEARNER
I20240502 21:27:09.483752 29725 consensus_queue.cc:260] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:09.487229 29725 ts_tablet_manager.cc:1398] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240502 21:27:09.488394 29093 tablet_copy_service.cc:342] P 9846ea932cb14dff98f20d67f1829791: Request end of tablet copy session 501feae33ef0484f956ca7a1e090ae01-ab35657135e44445aee4aa6961f70c27 received from {username='slave'} at 127.0.0.1:60746
I20240502 21:27:09.488510 29093 tablet_copy_service.cc:433] P 9846ea932cb14dff98f20d67f1829791: ending tablet copy session 501feae33ef0484f956ca7a1e090ae01-ab35657135e44445aee4aa6961f70c27 on tablet ab35657135e44445aee4aa6961f70c27 with peer 501feae33ef0484f956ca7a1e090ae01
I20240502 21:27:09.491693 29730 tablet_bootstrap.cc:492] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=88 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:09.492089 29730 tablet_bootstrap.cc:492] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01: Bootstrap complete.
I20240502 21:27:09.492367 29730 ts_tablet_manager.cc:1367] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01: Time spent bootstrapping tablet: real 0.035s user 0.029s sys 0.006s
I20240502 21:27:09.493063 29730 raft_consensus.cc:348] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:09.493314 29730 raft_consensus.cc:729] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 501feae33ef0484f956ca7a1e090ae01, State: Initialized, Role: LEARNER
I20240502 21:27:09.493494 29730 consensus_queue.cc:260] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:09.494038 29730 ts_tablet_manager.cc:1398] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:09.494699 29017 tablet_copy_service.cc:342] P 94f26f2c271544729391496f0fa81a58: Request end of tablet copy session 501feae33ef0484f956ca7a1e090ae01-31c63a7053fa45b69eba6d7f7e78139b received from {username='slave'} at 127.0.0.1:48068
I20240502 21:27:09.494805 29017 tablet_copy_service.cc:433] P 94f26f2c271544729391496f0fa81a58: ending tablet copy session 501feae33ef0484f956ca7a1e090ae01-31c63a7053fa45b69eba6d7f7e78139b on tablet 31c63a7053fa45b69eba6d7f7e78139b with peer 501feae33ef0484f956ca7a1e090ae01
I20240502 21:27:09.517426 29735 tablet_bootstrap.cc:492] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:09.517812 29735 tablet_bootstrap.cc:492] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap complete.
I20240502 21:27:09.518066 29735 ts_tablet_manager.cc:1367] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent bootstrapping tablet: real 0.046s user 0.037s sys 0.004s
I20240502 21:27:09.518698 29735 raft_consensus.cc:348] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:09.518965 29735 raft_consensus.cc:729] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c0c2fc14113d41c498fd6abcbbbd03c7, State: Initialized, Role: LEARNER
I20240502 21:27:09.519194 29735 consensus_queue.cc:260] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:09.520113 29735 ts_tablet_manager.cc:1398] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:09.520767 29093 tablet_copy_service.cc:342] P 9846ea932cb14dff98f20d67f1829791: Request end of tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-ccd9ffc92c5c47d088c530e919f2ff77 received from {username='slave'} at 127.0.0.1:60738
I20240502 21:27:09.520897 29093 tablet_copy_service.cc:433] P 9846ea932cb14dff98f20d67f1829791: ending tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-ccd9ffc92c5c47d088c530e919f2ff77 on tablet ccd9ffc92c5c47d088c530e919f2ff77 with peer c0c2fc14113d41c498fd6abcbbbd03c7
I20240502 21:27:09.533787 29477 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Advancing to term 3
I20240502 21:27:09.540683 29402 raft_consensus.cc:1203] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240502 21:27:09.809478 29477 raft_consensus.cc:1203] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.25->[2.26-2.26] Dedup: 2.26->[]
I20240502 21:27:09.889750 29402 raft_consensus.cc:1203] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240502 21:27:09.911566 29757 raft_consensus.cc:1050] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58: attempting to promote NON_VOTER 501feae33ef0484f956ca7a1e090ae01 to VOTER
I20240502 21:27:09.912238 29757 consensus_queue.cc:237] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:09.914633 29081 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:09.914629 29477 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 3 LEARNER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:09.915092 29157 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:09.915403 29760 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:09.915684 29741 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:09.915908 29779 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:09.917630 29477 raft_consensus.cc:1203] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20240502 21:27:09.919216 29741 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:09.920943 29081 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:09.920943 29476 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:09.925630 28902 catalog_manager.cc:5484] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:09.927814 29157 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:09.935725 29001 consensus_queue.cc:237] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:09.937409 29476 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 21:27:09.937606 29157 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 21:27:09.937906 29779 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:09.938201 29741 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:09.940407 29741 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:09.943394 28889 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af41f02157c94f55b2bc5090f566592e with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:09.943921 29476 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:09.944012 29157 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:09.944666 28895 catalog_manager.cc:5484] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index 29 to 30, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New cstate: current_term: 3 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:09.948897 29071 tablet_service.cc:1514] Processing DeleteTablet for tablet af41f02157c94f55b2bc5090f566592e with delete_type TABLET_DATA_TOMBSTONED (TS 9846ea932cb14dff98f20d67f1829791 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:51584
I20240502 21:27:09.949900 29784 tablet_replica.cc:330] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791: stopping tablet replica
I20240502 21:27:09.950105 29784 raft_consensus.cc:2229] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:09.950438 29784 raft_consensus.cc:2258] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:09.951361 29784 ts_tablet_manager.cc:1875] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:09.957271 29784 ts_tablet_manager.cc:1888] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20240502 21:27:09.957389 29784 log.cc:1196] T af41f02157c94f55b2bc5090f566592e P 9846ea932cb14dff98f20d67f1829791: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-1-root/wals/af41f02157c94f55b2bc5090f566592e
I20240502 21:27:09.957922 28890 catalog_manager.cc:4830] TS 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): tablet af41f02157c94f55b2bc5090f566592e (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:09.976876 29741 raft_consensus.cc:1050] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58: attempting to promote NON_VOTER 9c60776a34a54a09822291687d162a23 to VOTER
I20240502 21:27:09.977710 29741 consensus_queue.cc:237] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 26, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:09.979982 29157 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:09.980029 29552 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 LEARNER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:09.980592 29757 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:09.980919 29760 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:09.981700 29076 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:09.982134 29760 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:09.984071 29757 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 28 to 29, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:09.984648 29157 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:09.987659 29552 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:09.988610 28904 catalog_manager.cc:5484] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index 28 to 29, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:09.991842 29076 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:09.995851 29003 consensus_queue.cc:237] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:09.997707 29552 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 21:27:09.998255 29757 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:09.999086 29076 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 21:27:09.999509 29757 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:10.001250 29760 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:10.001886 29552 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:10.002032 29081 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:10.006444 28889 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54a27d02080d420382add870508aaad4 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:10.006286 28904 catalog_manager.cc:5484] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index 29 to 30, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New cstate: current_term: 3 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:10.010043 29138 tablet_service.cc:1514] Processing DeleteTablet for tablet 54a27d02080d420382add870508aaad4 with delete_type TABLET_DATA_TOMBSTONED (TS 5f1d764da7eb4f0c9fd4cabdca685d7b not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:34466
I20240502 21:27:10.010769 29789 tablet_replica.cc:330] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b: stopping tablet replica
I20240502 21:27:10.010954 29789 raft_consensus.cc:2229] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:10.011332 29789 raft_consensus.cc:2258] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:10.012215 29789 ts_tablet_manager.cc:1875] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:10.017105 29789 ts_tablet_manager.cc:1888] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20240502 21:27:10.017204 29789 log.cc:1196] T 54a27d02080d420382add870508aaad4 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-2-root/wals/54a27d02080d420382add870508aaad4
I20240502 21:27:10.017858 28890 catalog_manager.cc:4830] TS 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): tablet 54a27d02080d420382add870508aaad4 (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:10.043625 29791 raft_consensus.cc:1050] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791: attempting to promote NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 to VOTER
I20240502 21:27:10.044548 29791 consensus_queue.cc:237] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:10.047104 29402 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 LEARNER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 21:27:10.047070 29003 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 21:27:10.047189 29157 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 21:27:10.047703 29677 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:10.047978 29793 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:10.048365 29790 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:10.057780 29677 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.058840 29003 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.064263 29157 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.065102 28903 catalog_manager.cc:5484] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index 26 to 27, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.072119 29402 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.075917 29076 consensus_queue.cc:237] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:10.077915 29402 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 21:27:10.078325 29150 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 21:27:10.078706 29790 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:10.078934 29793 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:10.080940 29793 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.081624 29150 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.085237 28890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ccd9ffc92c5c47d088c530e919f2ff77 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:10.085325 29402 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.086205 28895 catalog_manager.cc:5484] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index 27 to 28, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New cstate: current_term: 1 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:10.090090 28994 tablet_service.cc:1514] Processing DeleteTablet for tablet ccd9ffc92c5c47d088c530e919f2ff77 with delete_type TABLET_DATA_TOMBSTONED (TS 94f26f2c271544729391496f0fa81a58 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:52236
I20240502 21:27:10.091228 29797 tablet_replica.cc:330] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58: stopping tablet replica
I20240502 21:27:10.091439 29797 raft_consensus.cc:2229] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:10.091856 29797 raft_consensus.cc:2258] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:10.092671 29797 ts_tablet_manager.cc:1875] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:10.097327 29797 ts_tablet_manager.cc:1888] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240502 21:27:10.097422 29797 log.cc:1196] T ccd9ffc92c5c47d088c530e919f2ff77 P 94f26f2c271544729391496f0fa81a58: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-0-root/wals/ccd9ffc92c5c47d088c530e919f2ff77
I20240502 21:27:10.098062 28889 catalog_manager.cc:4830] TS 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): tablet ccd9ffc92c5c47d088c530e919f2ff77 (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:10.259027 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:10.262305 29003 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
new_leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:52268
I20240502 21:27:10.262537 29003 raft_consensus.cc:595] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Received request to transfer leadership to TS 5f1d764da7eb4f0c9fd4cabdca685d7b
I20240502 21:27:10.263545 29003 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
new_leader_uuid: "501feae33ef0484f956ca7a1e090ae01"
from {username='slave'} at 127.0.0.1:52268
I20240502 21:27:10.263803 29003 raft_consensus.cc:595] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Received request to transfer leadership to TS 501feae33ef0484f956ca7a1e090ae01
I20240502 21:27:10.264786 29076 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ccd9ffc92c5c47d088c530e919f2ff77"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
mode: GRACEFUL
new_leader_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7"
from {username='slave'} at 127.0.0.1:60752
I20240502 21:27:10.264983 29076 raft_consensus.cc:595] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Received request to transfer leadership to TS c0c2fc14113d41c498fd6abcbbbd03c7
I20240502 21:27:10.265945 29076 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ee4eba5acb8d4af986a00f937de0270e"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
mode: GRACEFUL
new_leader_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:60752
I20240502 21:27:10.266110 29076 raft_consensus.cc:595] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Received request to transfer leadership to TS 94f26f2c271544729391496f0fa81a58
I20240502 21:27:10.266466 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240502 21:27:10.266691 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:10.282146 29791 raft_consensus.cc:1050] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791: attempting to promote NON_VOTER 501feae33ef0484f956ca7a1e090ae01 to VOTER
I20240502 21:27:10.283012 29791 consensus_queue.cc:237] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:10.285408 29150 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:10.285573 29001 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:10.285650 29476 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:10.286154 29677 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:10.286484 29793 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:10.287009 29793 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:10.299425 29791 raft_consensus.cc:2932] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index 26 to 27, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:10.299885 29476 raft_consensus.cc:2932] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:10.302390 29150 raft_consensus.cc:2932] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:10.305151 29001 raft_consensus.cc:2932] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:10.305567 28895 catalog_manager.cc:5484] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 reported cstate change: config changed from index 26 to 27, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:10.312474 29076 consensus_queue.cc:237] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:10.314057 29476 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 21:27:10.314268 29150 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 21:27:10.315680 29791 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:10.315933 29677 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:10.318182 29791 raft_consensus.cc:2932] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:10.318601 29476 raft_consensus.cc:2932] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:10.318572 29150 raft_consensus.cc:2932] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:10.322453 28890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab35657135e44445aee4aa6961f70c27 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:10.323379 28902 catalog_manager.cc:5484] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index 27 to 28, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New cstate: current_term: 2 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:10.327540 28994 tablet_service.cc:1514] Processing DeleteTablet for tablet ab35657135e44445aee4aa6961f70c27 with delete_type TABLET_DATA_TOMBSTONED (TS 94f26f2c271544729391496f0fa81a58 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:52236
I20240502 21:27:10.327857 29797 tablet_replica.cc:330] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58: stopping tablet replica
I20240502 21:27:10.328049 29797 raft_consensus.cc:2229] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:10.328334 29797 raft_consensus.cc:2258] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:10.329180 29797 ts_tablet_manager.cc:1875] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:10.334852 29797 ts_tablet_manager.cc:1888] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.27
I20240502 21:27:10.334960 29797 log.cc:1196] T ab35657135e44445aee4aa6961f70c27 P 94f26f2c271544729391496f0fa81a58: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-0-root/wals/ab35657135e44445aee4aa6961f70c27
I20240502 21:27:10.335736 28889 catalog_manager.cc:4830] TS 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): tablet ab35657135e44445aee4aa6961f70c27 (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:10.389323 29791 raft_consensus.cc:979] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791: : Instructing follower 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:10.389508 29677 raft_consensus.cc:1067] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Signalling peer 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:10.390129 29003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ee4eba5acb8d4af986a00f937de0270e"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:52252
I20240502 21:27:10.390295 29003 raft_consensus.cc:482] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:10.390389 29003 raft_consensus.cc:3037] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:10.392385 29003 raft_consensus.cc:504] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:10.393026 29003 leader_election.cc:288] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 2 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:10.393363 29081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4eba5acb8d4af986a00f937de0270e" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:10.393476 29150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4eba5acb8d4af986a00f937de0270e" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:10.393576 29081 raft_consensus.cc:3032] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:10.393664 29150 raft_consensus.cc:3037] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:10.393692 29081 raft_consensus.cc:729] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:10.394053 29081 consensus_queue.cc:260] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:10.394508 29081 raft_consensus.cc:3037] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:10.395932 29150 raft_consensus.cc:2454] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 2.
I20240502 21:27:10.396404 28972 leader_election.cc:302] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:27:10.396615 29081 raft_consensus.cc:2454] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 2.
I20240502 21:27:10.396673 29757 raft_consensus.cc:2781] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:10.396787 29757 raft_consensus.cc:686] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:10.397063 29757 consensus_queue.cc:237] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:10.400327 28904 catalog_manager.cc:5484] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 1 to 2, leader changed from 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 2 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:10.479583 29779 raft_consensus.cc:979] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58: : Instructing follower 501feae33ef0484f956ca7a1e090ae01 to start an election
I20240502 21:27:10.479756 29779 raft_consensus.cc:1067] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Signalling peer 501feae33ef0484f956ca7a1e090ae01 to start an election
I20240502 21:27:10.480381 29476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e"
dest_uuid: "501feae33ef0484f956ca7a1e090ae01"
from {username='slave'} at 127.0.0.1:33708
I20240502 21:27:10.480599 29476 raft_consensus.cc:482] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:10.480715 29476 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:10.483497 29476 raft_consensus.cc:504] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:10.484473 29476 leader_election.cc:288] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [CANDIDATE]: Term 4 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:10.484624 29003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e" candidate_uuid: "501feae33ef0484f956ca7a1e090ae01" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:10.484719 29150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af41f02157c94f55b2bc5090f566592e" candidate_uuid: "501feae33ef0484f956ca7a1e090ae01" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:10.484805 29003 raft_consensus.cc:3032] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:10.484875 29003 raft_consensus.cc:729] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:10.484906 29150 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:10.485113 29003 consensus_queue.cc:260] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:10.485385 29003 raft_consensus.cc:3037] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:10.487031 29150 raft_consensus.cc:2454] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501feae33ef0484f956ca7a1e090ae01 in term 4.
I20240502 21:27:10.487340 29003 raft_consensus.cc:2454] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501feae33ef0484f956ca7a1e090ae01 in term 4.
I20240502 21:27:10.487408 29442 leader_election.cc:302] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 501feae33ef0484f956ca7a1e090ae01, 5f1d764da7eb4f0c9fd4cabdca685d7b; no voters:
I20240502 21:27:10.487692 29771 raft_consensus.cc:2781] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:10.488157 29771 raft_consensus.cc:686] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 4 LEADER]: Becoming Leader. State: Replica: 501feae33ef0484f956ca7a1e090ae01, State: Running, Role: LEADER
I20240502 21:27:10.488476 29771 consensus_queue.cc:237] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:10.490926 28904 catalog_manager.cc:5484] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 reported cstate change: term changed from 3 to 4, leader changed from 94f26f2c271544729391496f0fa81a58 (127.25.16.65) to 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69). New cstate: current_term: 4 leader_uuid: "501feae33ef0484f956ca7a1e090ae01" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:10.538292 29804 raft_consensus.cc:1050] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: attempting to promote NON_VOTER 501feae33ef0484f956ca7a1e090ae01 to VOTER
W20240502 21:27:10.538420 29804 raft_consensus.cc:1054] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: Unable to promote non-voter 501feae33ef0484f956ca7a1e090ae01: Service unavailable: leader transfer in progress
I20240502 21:27:10.574122 29677 raft_consensus.cc:979] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791: : Instructing follower c0c2fc14113d41c498fd6abcbbbd03c7 to start an election
I20240502 21:27:10.574295 29791 raft_consensus.cc:1067] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Signalling peer c0c2fc14113d41c498fd6abcbbbd03c7 to start an election
I20240502 21:27:10.574818 29402 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ccd9ffc92c5c47d088c530e919f2ff77"
dest_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7"
from {username='slave'} at 127.0.0.1:59376
I20240502 21:27:10.575047 29402 raft_consensus.cc:482] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:10.575145 29402 raft_consensus.cc:3037] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:10.577349 29402 raft_consensus.cc:504] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:10.578225 29402 leader_election.cc:288] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [CANDIDATE]: Term 2 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:10.578603 29076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd9ffc92c5c47d088c530e919f2ff77" candidate_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:10.587502 29076 raft_consensus.cc:3032] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:10.587713 29076 raft_consensus.cc:729] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:10.588063 29076 consensus_queue.cc:260] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:10.588568 29076 raft_consensus.cc:3037] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:10.589656 29150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd9ffc92c5c47d088c530e919f2ff77" candidate_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:10.589826 29150 raft_consensus.cc:3037] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:10.592355 29150 raft_consensus.cc:2454] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0c2fc14113d41c498fd6abcbbbd03c7 in term 2.
I20240502 21:27:10.592355 29076 raft_consensus.cc:2454] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0c2fc14113d41c498fd6abcbbbd03c7 in term 2.
I20240502 21:27:10.592761 29367 leader_election.cc:302] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, c0c2fc14113d41c498fd6abcbbbd03c7; no voters:
I20240502 21:27:10.593323 29807 raft_consensus.cc:2781] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:10.593720 29807 raft_consensus.cc:686] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEADER]: Becoming Leader. State: Replica: c0c2fc14113d41c498fd6abcbbbd03c7, State: Running, Role: LEADER
I20240502 21:27:10.593995 29807 consensus_queue.cc:237] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:10.596333 28904 catalog_manager.cc:5484] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 reported cstate change: term changed from 1 to 2, leader changed from 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) to c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68). New cstate: current_term: 2 leader_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:10.619975 29757 raft_consensus.cc:979] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: : Instructing follower 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:10.620122 29757 raft_consensus.cc:1067] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Signalling peer 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:10.620692 29150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:34468
I20240502 21:27:10.620862 29150 raft_consensus.cc:482] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:10.620939 29150 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:10.623680 29150 raft_consensus.cc:504] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:10.624737 29076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:10.624930 29076 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:10.624899 29003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:10.625097 29003 raft_consensus.cc:3032] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:10.625170 29003 raft_consensus.cc:729] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:10.625444 29003 consensus_queue.cc:260] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:10.625768 29150 leader_election.cc:288] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 3 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:10.625803 29003 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:10.627194 29076 raft_consensus.cc:2454] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 3.
I20240502 21:27:10.627554 29122 leader_election.cc:302] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:10.627768 29003 raft_consensus.cc:2454] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 3.
I20240502 21:27:10.627801 29743 raft_consensus.cc:2781] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:10.641121 29743 raft_consensus.cc:686] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:10.641469 29743 consensus_queue.cc:237] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:10.644472 28904 catalog_manager.cc:5484] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 2 to 3, leader changed from 94f26f2c271544729391496f0fa81a58 (127.25.16.65) to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 3 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:10.814693 29150 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:10.815142 29757 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:10.819857 29076 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:10.821805 29779 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:10.837232 29402 raft_consensus.cc:3037] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 1 LEARNER]: Advancing to term 2
I20240502 21:27:10.840150 29402 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEARNER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:10.840899 29779 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:10.843791 29779 raft_consensus.cc:1050] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58: attempting to promote NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 to VOTER
I20240502 21:27:10.844465 29779 consensus_queue.cc:237] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:10.846812 29150 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:10.847000 29402 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEARNER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:10.847549 29076 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:10.847486 29757 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:10.847939 29757 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:10.848572 29757 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:10.850590 29804 raft_consensus.cc:2932] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index 25 to 27, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.851272 29076 raft_consensus.cc:2932] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 25 to 27, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.855113 29402 raft_consensus.cc:2932] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 25 to 27, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.857064 28903 catalog_manager.cc:5484] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index 25 to 27, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.858781 29150 raft_consensus.cc:2932] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 25 to 27, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.863605 29003 consensus_queue.cc:237] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:10.865512 29402 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 21:27:10.865562 29076 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 21:27:10.866070 29779 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:10.866341 29820 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:10.868834 29804 raft_consensus.cc:2932] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.871766 28889 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee4eba5acb8d4af986a00f937de0270e with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:10.872396 29402 raft_consensus.cc:2932] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.873070 28902 catalog_manager.cc:5484] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index 27 to 28, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New cstate: current_term: 2 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:10.874990 29076 raft_consensus.cc:2932] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:10.876972 29138 tablet_service.cc:1514] Processing DeleteTablet for tablet ee4eba5acb8d4af986a00f937de0270e with delete_type TABLET_DATA_TOMBSTONED (TS 5f1d764da7eb4f0c9fd4cabdca685d7b not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:34466
I20240502 21:27:10.877741 29821 tablet_replica.cc:330] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b: stopping tablet replica
I20240502 21:27:10.878043 29821 raft_consensus.cc:2229] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:10.878636 29821 raft_consensus.cc:2258] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:10.880839 29821 ts_tablet_manager.cc:1875] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:10.885985 29821 ts_tablet_manager.cc:1888] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.27
I20240502 21:27:10.886097 29821 log.cc:1196] T ee4eba5acb8d4af986a00f937de0270e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-2-root/wals/ee4eba5acb8d4af986a00f937de0270e
I20240502 21:27:10.886655 28890 catalog_manager.cc:4830] TS 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): tablet ee4eba5acb8d4af986a00f937de0270e (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:10.914883 29003 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Refusing update from remote peer 501feae33ef0484f956ca7a1e090ae01: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 21:27:10.915422 29771 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:10.919145 29150 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Refusing update from remote peer 501feae33ef0484f956ca7a1e090ae01: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 21:27:10.920102 29771 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:10.986933 29157 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer c0c2fc14113d41c498fd6abcbbbd03c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 21:27:10.987474 29807 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:10.991461 29076 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer c0c2fc14113d41c498fd6abcbbbd03c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 21:27:10.992640 29808 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:11.063213 29003 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 21:27:11.063745 29743 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:11.069025 29476 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Advancing to term 3
I20240502 21:27:11.072397 29476 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 3 LEARNER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 21:27:11.072786 29076 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 21:27:11.072969 29825 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:11.074101 29825 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:11.087692 29826 raft_consensus.cc:1050] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: attempting to promote NON_VOTER 501feae33ef0484f956ca7a1e090ae01 to VOTER
I20240502 21:27:11.088580 29826 consensus_queue.cc:237] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:11.090664 29003 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:11.090824 29476 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 3 LEARNER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:11.091167 29825 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:11.090664 29076 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:11.091409 29826 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:11.092002 29827 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:11.094293 29825 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:11.094990 29476 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:11.096987 29076 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:11.098683 29003 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:11.101104 28904 catalog_manager.cc:5484] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 reported cstate change: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:11.107161 29150 consensus_queue.cc:237] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:11.108656 29476 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 21:27:11.108659 29003 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 21:27:11.109125 29826 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:11.109381 29825 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:11.111560 29743 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:11.112413 29003 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:11.115803 28890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 31c63a7053fa45b69eba6d7f7e78139b with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:11.116701 28902 catalog_manager.cc:5484] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index 29 to 30, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New cstate: current_term: 3 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:11.117131 29476 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:11.120033 29071 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c63a7053fa45b69eba6d7f7e78139b with delete_type TABLET_DATA_TOMBSTONED (TS 9846ea932cb14dff98f20d67f1829791 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:51584
I20240502 21:27:11.120918 29830 tablet_replica.cc:330] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791: stopping tablet replica
I20240502 21:27:11.121238 29830 raft_consensus.cc:2229] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:11.121459 29830 raft_consensus.cc:2258] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:11.122177 29830 ts_tablet_manager.cc:1875] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:11.126405 29830 ts_tablet_manager.cc:1888] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20240502 21:27:11.126514 29830 log.cc:1196] T 31c63a7053fa45b69eba6d7f7e78139b P 9846ea932cb14dff98f20d67f1829791: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-1-root/wals/31c63a7053fa45b69eba6d7f7e78139b
I20240502 21:27:11.127132 28890 catalog_manager.cc:4830] TS 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): tablet 31c63a7053fa45b69eba6d7f7e78139b (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:12.122717 29157 consensus_queue.cc:237] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:12.125087 29003 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 21:27:12.128156 29834 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 21:27:12.128264 29076 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240502 21:27:12.128840 29825 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
W20240502 21:27:12.136142 29121 consensus_peers.cc:485] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b -> Peer 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791): Couldn't send request to peer 9c60776a34a54a09822291687d162a23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ce57dd3a92645839765ba1ce6609430. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:12.136833 29825 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:12.137442 29076 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:12.140511 29003 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:12.142547 28902 catalog_manager.cc:5484] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index -1 to 25, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New cstate: current_term: 1 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:12.142938 29081 consensus_queue.cc:237] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
W20240502 21:27:12.145982 29046 consensus_peers.cc:485] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 -> Peer c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499): Couldn't send request to peer c0c2fc14113d41c498fd6abcbbbd03c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d08f3643db34c8ba4d4391b630be366. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:12.147037 29003 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:12.147307 29157 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:12.147861 29843 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:12.148154 29677 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:12.151422 29677 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:12.151811 29157 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:12.152138 29003 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:12.157027 28903 catalog_manager.cc:5484] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index -1 to 27, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New cstate: current_term: 2 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:12.157357 29157 consensus_queue.cc:237] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:12.159617 29003 raft_consensus.cc:1261] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
W20240502 21:27:12.159763 29122 consensus_peers.cc:485] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b -> Peer 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297): Couldn't send request to peer 501feae33ef0484f956ca7a1e090ae01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5010a09a5243452dad7cd9b90535a229. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:12.160102 29834 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:12.160315 29081 raft_consensus.cc:1261] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:12.160773 29834 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:12.163146 29835 raft_consensus.cc:2932] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:12.164091 29081 raft_consensus.cc:2932] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:12.164822 29003 raft_consensus.cc:2932] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:12.168752 29001 consensus_queue.cc:237] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:12.169158 28895 catalog_manager.cc:5484] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index -1 to 27, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New cstate: current_term: 2 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:12.171837 29552 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 21:27:12.172313 29081 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 21:27:12.172734 29854 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:12.173224 29779 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20240502 21:27:12.173491 28970 consensus_peers.cc:485] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 -> Peer c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499): Couldn't send request to peer c0c2fc14113d41c498fd6abcbbbd03c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54a27d02080d420382add870508aaad4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:12.177080 29779 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:12.180155 29552 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:12.181406 29003 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:12.186345 28902 catalog_manager.cc:5484] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 reported cstate change: config changed from index 30 to 31, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New cstate: current_term: 3 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:12.191850 29081 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:12.192241 29157 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 21:27:12.192801 29855 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:12.193225 29076 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
W20240502 21:27:12.194355 28971 consensus_peers.cc:485] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 -> Peer 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791): Couldn't send request to peer 9c60776a34a54a09822291687d162a23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea5a9bdae3f3459f915eaf3bdd3d346c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:12.194800 29855 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:12.195899 29779 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:12.199520 29157 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:12.200364 29081 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:12.204511 28903 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index -1 to 26, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New cstate: current_term: 1 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:12.226639 29476 consensus_queue.cc:237] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:12.229099 29003 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Refusing update from remote peer 501feae33ef0484f956ca7a1e090ae01: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 21:27:12.229666 29865 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:12.229903 29157 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Refusing update from remote peer 501feae33ef0484f956ca7a1e090ae01: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 21:27:12.230336 29865 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:12.232563 29864 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 30 to 32, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:12.233139 29003 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 30 to 32, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:12.233269 29157 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 30 to 32, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
W20240502 21:27:12.235062 29440 consensus_peers.cc:485] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 -> Peer c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499): Couldn't send request to peer c0c2fc14113d41c498fd6abcbbbd03c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af41f02157c94f55b2bc5090f566592e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:12.237826 28902 catalog_manager.cc:5484] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index 30 to 32, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New cstate: current_term: 4 leader_uuid: "501feae33ef0484f956ca7a1e090ae01" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:12.266943 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:12.269744 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5010a09a5243452dad7cd9b90535a229"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
new_leader_uuid: "9846ea932cb14dff98f20d67f1829791"
from {username='slave'} at 127.0.0.1:42850
I20240502 21:27:12.269973 29157 raft_consensus.cc:595] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Received request to transfer leadership to TS 9846ea932cb14dff98f20d67f1829791
I20240502 21:27:12.270819 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "93477dec2c574c49a0bc9a7f022ad7fa"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
new_leader_uuid: "9846ea932cb14dff98f20d67f1829791"
from {username='slave'} at 127.0.0.1:42850
I20240502 21:27:12.270987 29157 raft_consensus.cc:595] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Received request to transfer leadership to TS 9846ea932cb14dff98f20d67f1829791
I20240502 21:27:12.271826 29003 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
new_leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:52268
I20240502 21:27:12.271989 29003 raft_consensus.cc:595] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Received request to transfer leadership to TS 5f1d764da7eb4f0c9fd4cabdca685d7b
I20240502 21:27:12.272382 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 21:27:12.272593 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:12.598655 29871 ts_tablet_manager.cc:902] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01: Initiating tablet copy from peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:12.600008 29871 tablet_copy_client.cc:308] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.67:41785
I20240502 21:27:12.600580 29167 tablet_copy_service.cc:140] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Received BeginTabletCopySession request for tablet 5010a09a5243452dad7cd9b90535a229 from peer 501feae33ef0484f956ca7a1e090ae01 ({username='slave'} at 127.0.0.1:42866)
I20240502 21:27:12.600697 29167 tablet_copy_service.cc:161] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Beginning new tablet copy session on tablet 5010a09a5243452dad7cd9b90535a229 from peer 501feae33ef0484f956ca7a1e090ae01 at {username='slave'} at 127.0.0.1:42866: session id = 501feae33ef0484f956ca7a1e090ae01-5010a09a5243452dad7cd9b90535a229
I20240502 21:27:12.603155 29167 tablet_copy_source_session.cc:205] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:12.604211 29871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5010a09a5243452dad7cd9b90535a229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.604925 29876 ts_tablet_manager.cc:902] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23: Initiating tablet copy from peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:12.610172 29880 ts_tablet_manager.cc:902] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7: Initiating tablet copy from peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159)
I20240502 21:27:12.611483 29871 tablet_copy_client.cc:783] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:12.611682 29871 tablet_copy_client.cc:647] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:12.613837 29875 tablet_copy_client.cc:1002] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:12.614827 29855 raft_consensus.cc:979] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58: : Instructing follower 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:12.614953 29855 raft_consensus.cc:1067] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Signalling peer 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:12.615489 29157 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:34468
I20240502 21:27:12.615667 29157 raft_consensus.cc:482] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:12.615746 29157 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:12.616607 29876 tablet_copy_client.cc:308] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.67:41785
I20240502 21:27:12.617240 29167 tablet_copy_service.cc:140] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Received BeginTabletCopySession request for tablet 7ce57dd3a92645839765ba1ce6609430 from peer 9c60776a34a54a09822291687d162a23 ({username='slave'} at 127.0.0.1:42872)
I20240502 21:27:12.617367 29167 tablet_copy_service.cc:161] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Beginning new tablet copy session on tablet 7ce57dd3a92645839765ba1ce6609430 from peer 9c60776a34a54a09822291687d162a23 at {username='slave'} at 127.0.0.1:42872: session id = 9c60776a34a54a09822291687d162a23-7ce57dd3a92645839765ba1ce6609430
I20240502 21:27:12.617884 29880 tablet_copy_client.cc:308] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.65:33159
I20240502 21:27:12.618753 29157 raft_consensus.cc:504] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:12.619832 29167 tablet_copy_source_session.cc:205] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:12.620234 29003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:12.620424 29003 raft_consensus.cc:3032] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:12.620463 29081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:12.620661 29081 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:12.623844 29081 raft_consensus.cc:2454] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 2.
I20240502 21:27:12.620486 29003 raft_consensus.cc:729] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:12.624842 29003 consensus_queue.cc:260] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:12.625299 29003 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:12.628108 29003 raft_consensus.cc:2454] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 2.
I20240502 21:27:12.628556 29122 leader_election.cc:302] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:12.628813 29835 raft_consensus.cc:2781] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:12.628955 29835 raft_consensus.cc:686] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:12.629187 29835 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:12.629437 29876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce57dd3a92645839765ba1ce6609430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.629693 29157 leader_election.cc:288] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 2 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:12.631922 28903 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 1 to 2, leader changed from 94f26f2c271544729391496f0fa81a58 (127.25.16.65) to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 2 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:12.634913 29017 tablet_copy_service.cc:140] P 94f26f2c271544729391496f0fa81a58: Received BeginTabletCopySession request for tablet 54a27d02080d420382add870508aaad4 from peer c0c2fc14113d41c498fd6abcbbbd03c7 ({username='slave'} at 127.0.0.1:54710)
I20240502 21:27:12.635041 29017 tablet_copy_service.cc:161] P 94f26f2c271544729391496f0fa81a58: Beginning new tablet copy session on tablet 54a27d02080d420382add870508aaad4 from peer c0c2fc14113d41c498fd6abcbbbd03c7 at {username='slave'} at 127.0.0.1:54710: session id = c0c2fc14113d41c498fd6abcbbbd03c7-54a27d02080d420382add870508aaad4
I20240502 21:27:12.635313 29876 tablet_copy_client.cc:783] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:12.635576 29876 tablet_copy_client.cc:647] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:12.637091 29885 tablet_copy_client.cc:1002] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:12.637806 29017 tablet_copy_source_session.cc:205] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:12.638700 29880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a27d02080d420382add870508aaad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.639036 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34466
I20240502 21:27:12.639199 29157 raft_consensus.cc:595] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Received request to transfer leadership
I20240502 21:27:12.643833 29880 tablet_copy_client.cc:783] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:12.644037 29880 tablet_copy_client.cc:647] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:12.645335 29884 tablet_copy_client.cc:1002] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:12.646843 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34466
I20240502 21:27:12.646975 29157 raft_consensus.cc:595] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Received request to transfer leadership
W20240502 21:27:12.647245 28890 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for ea5a9bdae3f3459f915eaf3bdd3d346c already in progress [suppressed 8 similar messages]
I20240502 21:27:12.648079 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34466
I20240502 21:27:12.648193 29157 raft_consensus.cc:595] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Received request to transfer leadership
W20240502 21:27:12.648586 28953 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for ea5a9bdae3f3459f915eaf3bdd3d346c already in progress
W20240502 21:27:12.648653 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for ea5a9bdae3f3459f915eaf3bdd3d346c already in progress
I20240502 21:27:12.676502 29889 ts_tablet_manager.cc:902] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7: Initiating tablet copy from peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:12.678038 29889 tablet_copy_client.cc:308] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.66:46109
I20240502 21:27:12.678529 29093 tablet_copy_service.cc:140] P 9846ea932cb14dff98f20d67f1829791: Received BeginTabletCopySession request for tablet 6d08f3643db34c8ba4d4391b630be366 from peer c0c2fc14113d41c498fd6abcbbbd03c7 ({username='slave'} at 127.0.0.1:60738)
I20240502 21:27:12.678629 29093 tablet_copy_service.cc:161] P 9846ea932cb14dff98f20d67f1829791: Beginning new tablet copy session on tablet 6d08f3643db34c8ba4d4391b630be366 from peer c0c2fc14113d41c498fd6abcbbbd03c7 at {username='slave'} at 127.0.0.1:60738: session id = c0c2fc14113d41c498fd6abcbbbd03c7-6d08f3643db34c8ba4d4391b630be366
I20240502 21:27:12.680950 29093 tablet_copy_source_session.cc:205] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:12.681911 29889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d08f3643db34c8ba4d4391b630be366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.687023 29889 tablet_copy_client.cc:783] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:12.687232 29889 tablet_copy_client.cc:647] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:12.688650 29893 tablet_copy_client.cc:1002] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:12.723265 29894 ts_tablet_manager.cc:902] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7: Initiating tablet copy from peer 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297)
I20240502 21:27:12.724821 29894 tablet_copy_client.cc:308] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.69:35297
I20240502 21:27:12.730324 29488 tablet_copy_service.cc:140] P 501feae33ef0484f956ca7a1e090ae01: Received BeginTabletCopySession request for tablet af41f02157c94f55b2bc5090f566592e from peer c0c2fc14113d41c498fd6abcbbbd03c7 ({username='slave'} at 127.0.0.1:60412)
I20240502 21:27:12.730466 29488 tablet_copy_service.cc:161] P 501feae33ef0484f956ca7a1e090ae01: Beginning new tablet copy session on tablet af41f02157c94f55b2bc5090f566592e from peer c0c2fc14113d41c498fd6abcbbbd03c7 at {username='slave'} at 127.0.0.1:60412: session id = c0c2fc14113d41c498fd6abcbbbd03c7-af41f02157c94f55b2bc5090f566592e
I20240502 21:27:12.732828 29488 tablet_copy_source_session.cc:205] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:12.734016 29894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af41f02157c94f55b2bc5090f566592e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:12.734366 29835 raft_consensus.cc:979] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b: : Instructing follower 9846ea932cb14dff98f20d67f1829791 to start an election
I20240502 21:27:12.734508 29835 raft_consensus.cc:1067] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Signalling peer 9846ea932cb14dff98f20d67f1829791 to start an election
I20240502 21:27:12.735122 29081 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "93477dec2c574c49a0bc9a7f022ad7fa"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
from {username='slave'} at 127.0.0.1:51598
I20240502 21:27:12.735294 29081 raft_consensus.cc:482] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:12.735380 29081 raft_consensus.cc:3037] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:12.738287 29081 raft_consensus.cc:504] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:12.739320 29081 leader_election.cc:288] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 2 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:12.739389 29150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93477dec2c574c49a0bc9a7f022ad7fa" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:12.739571 29150 raft_consensus.cc:3032] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:12.739650 29150 raft_consensus.cc:729] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:12.739866 29150 consensus_queue.cc:260] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:12.740134 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93477dec2c574c49a0bc9a7f022ad7fa" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:12.740312 29001 raft_consensus.cc:3037] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:12.740237 29150 raft_consensus.cc:3037] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:12.740674 29894 tablet_copy_client.cc:783] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:12.740873 29894 tablet_copy_client.cc:647] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:12.742130 29898 tablet_copy_client.cc:1002] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:12.743186 29150 raft_consensus.cc:2454] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 2.
I20240502 21:27:12.743186 29001 raft_consensus.cc:2454] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 2.
I20240502 21:27:12.743551 29047 leader_election.cc:302] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:12.743789 29849 raft_consensus.cc:2781] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:12.744190 29849 raft_consensus.cc:686] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Becoming Leader. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:12.744473 29849 consensus_queue.cc:237] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:12.746958 28903 catalog_manager.cc:5484] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 reported cstate change: term changed from 1 to 2, leader changed from 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) to 9846ea932cb14dff98f20d67f1829791 (127.25.16.66). New cstate: current_term: 2 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:12.784543 29825 raft_consensus.cc:979] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b: : Instructing follower 9846ea932cb14dff98f20d67f1829791 to start an election
I20240502 21:27:12.784706 29825 raft_consensus.cc:1067] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Signalling peer 9846ea932cb14dff98f20d67f1829791 to start an election
I20240502 21:27:12.785274 29081 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5010a09a5243452dad7cd9b90535a229"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
from {username='slave'} at 127.0.0.1:51598
I20240502 21:27:12.785429 29081 raft_consensus.cc:482] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:12.785512 29081 raft_consensus.cc:3037] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:12.787581 29081 raft_consensus.cc:504] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:12.788141 29081 leader_election.cc:288] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 3 election: Requested vote from peers 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785), 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159)
I20240502 21:27:12.788520 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5010a09a5243452dad7cd9b90535a229" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:12.788724 29001 raft_consensus.cc:3037] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:12.790869 29001 raft_consensus.cc:2454] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 3.
I20240502 21:27:12.788520 29150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5010a09a5243452dad7cd9b90535a229" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:12.791134 29150 raft_consensus.cc:3032] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:12.791186 29150 raft_consensus.cc:729] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:12.791215 29047 leader_election.cc:302] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:12.791472 29849 raft_consensus.cc:2781] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:12.791386 29150 consensus_queue.cc:260] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:12.791582 29849 raft_consensus.cc:686] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 3 LEADER]: Becoming Leader. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:12.791779 29150 raft_consensus.cc:3037] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:12.791824 29849 consensus_queue.cc:237] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:12.794363 29150 raft_consensus.cc:2454] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 3.
I20240502 21:27:12.794376 28903 catalog_manager.cc:5484] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: term changed from 2 to 3, leader changed from 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) to 9846ea932cb14dff98f20d67f1829791 (127.25.16.66). New cstate: current_term: 3 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:13.136832 29001 raft_consensus.cc:1261] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:13.137471 29849 consensus_queue.cc:1035] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:13.143361 29150 raft_consensus.cc:1261] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240502 21:27:13.143926 29849 consensus_queue.cc:1035] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:13.172319 29076 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:13.172803 29835 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:13.174489 29825 raft_consensus.cc:979] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b: : Instructing follower 9846ea932cb14dff98f20d67f1829791 to start an election
I20240502 21:27:13.174608 29825 raft_consensus.cc:1067] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Signalling peer 9846ea932cb14dff98f20d67f1829791 to start an election
I20240502 21:27:13.175702 29001 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240502 21:27:13.175974 29081 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
from {username='slave'} at 127.0.0.1:51598
W20240502 21:27:13.176164 29121 consensus_peers.cc:485] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b -> Peer 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791): Couldn't send request to peer 9c60776a34a54a09822291687d162a23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea5a9bdae3f3459f915eaf3bdd3d346c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.176162 29081 raft_consensus.cc:482] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:13.176398 29081 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:13.178350 29081 raft_consensus.cc:504] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:13.178628 29076 raft_consensus.cc:1226] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Rejecting Update request from peer 5f1d764da7eb4f0c9fd4cabdca685d7b for earlier term 2. Current term is 3. Ops: []
I20240502 21:27:13.178854 29081 leader_election.cc:288] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 3 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:13.179122 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:13.179206 29825 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:13.179284 29001 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:13.179605 29825 consensus_queue.cc:1046] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: INVALID_TERM, Last received: 2.27, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:13.179874 29901 raft_consensus.cc:3032] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:13.179963 29901 raft_consensus.cc:729] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:13.180212 29901 consensus_queue.cc:260] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:13.180428 29150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:13.180652 29901 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:13.181921 29001 raft_consensus.cc:2454] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 3.
I20240502 21:27:13.182093 29003 raft_consensus.cc:1226] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Rejecting Update request from peer 5f1d764da7eb4f0c9fd4cabdca685d7b for earlier term 2. Current term is 3. Ops: [2.27-2.27]
I20240502 21:27:13.182665 29047 leader_election.cc:302] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:13.182930 29899 raft_consensus.cc:2781] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:13.183034 29899 raft_consensus.cc:686] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 3 LEADER]: Becoming Leader. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:13.183246 29899 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:13.186118 29150 raft_consensus.cc:2454] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 3.
I20240502 21:27:13.187008 28895 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 reported cstate change: term changed from 2 to 3, leader changed from 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) to 9846ea932cb14dff98f20d67f1829791 (127.25.16.66). New cstate: current_term: 3 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:13.205947 29157 raft_consensus.cc:1261] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 21:27:13.206362 29905 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:13.209437 29001 raft_consensus.cc:1261] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 21:27:13.209897 29899 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240502 21:27:13.210659 29048 consensus_peers.cc:485] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 -> Peer 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297): Couldn't send request to peer 501feae33ef0484f956ca7a1e090ae01. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.572121 29003 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 21:27:13.572671 29905 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
W20240502 21:27:13.577039 29047 consensus_peers.cc:485] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 -> Peer 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791): Couldn't send request to peer 9c60776a34a54a09822291687d162a23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea5a9bdae3f3459f915eaf3bdd3d346c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.577319 29150 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 21:27:13.577749 29899 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:13.956853 29920 ts_tablet_manager.cc:902] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23: Initiating tablet copy from peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:13.958146 29920 tablet_copy_client.cc:308] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.66:46109
I20240502 21:27:13.958649 29093 tablet_copy_service.cc:140] P 9846ea932cb14dff98f20d67f1829791: Received BeginTabletCopySession request for tablet ea5a9bdae3f3459f915eaf3bdd3d346c from peer 9c60776a34a54a09822291687d162a23 ({username='slave'} at 127.0.0.1:60728)
I20240502 21:27:13.958788 29093 tablet_copy_service.cc:161] P 9846ea932cb14dff98f20d67f1829791: Beginning new tablet copy session on tablet ea5a9bdae3f3459f915eaf3bdd3d346c from peer 9c60776a34a54a09822291687d162a23 at {username='slave'} at 127.0.0.1:60728: session id = 9c60776a34a54a09822291687d162a23-ea5a9bdae3f3459f915eaf3bdd3d346c
I20240502 21:27:13.961663 29093 tablet_copy_source_session.cc:205] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:13.962615 29920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5a9bdae3f3459f915eaf3bdd3d346c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:13.967980 29920 tablet_copy_client.cc:783] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:13.968647 29920 tablet_copy_client.cc:647] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:13.969825 29924 tablet_copy_client.cc:1002] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:14.272912 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:14.275874 29076 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
mode: GRACEFUL
new_leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:60752
I20240502 21:27:14.276077 29076 raft_consensus.cc:595] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Received request to transfer leadership to TS 5f1d764da7eb4f0c9fd4cabdca685d7b
I20240502 21:27:14.277048 29076 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
mode: GRACEFUL
new_leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:60752
I20240502 21:27:14.277204 29076 raft_consensus.cc:595] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Received request to transfer leadership to TS 5f1d764da7eb4f0c9fd4cabdca685d7b
I20240502 21:27:14.278055 29076 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
mode: GRACEFUL
new_leader_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:60752
I20240502 21:27:14.278189 29076 raft_consensus.cc:595] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 3 LEADER]: Received request to transfer leadership to TS 94f26f2c271544729391496f0fa81a58
I20240502 21:27:14.278512 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 21:27:14.278704 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:14.395295 29916 raft_consensus.cc:979] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791: : Instructing follower 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:14.395445 29916 raft_consensus.cc:1067] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 3 LEADER]: Signalling peer 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:14.395960 29003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:52252
I20240502 21:27:14.396128 29003 raft_consensus.cc:482] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:14.396226 29003 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:14.398391 29003 raft_consensus.cc:504] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:14.398988 29003 leader_election.cc:288] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 4 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:14.399274 29081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:14.399425 29081 raft_consensus.cc:3032] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:14.399487 29081 raft_consensus.cc:729] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:14.399497 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:14.399674 29157 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:14.399717 29081 consensus_queue.cc:260] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:14.400074 29081 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:14.401530 29157 raft_consensus.cc:2454] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 4.
I20240502 21:27:14.401778 29081 raft_consensus.cc:2454] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 4.
I20240502 21:27:14.401926 28972 leader_election.cc:302] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:27:14.402221 29909 raft_consensus.cc:2781] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:14.402591 29909 raft_consensus.cc:686] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:14.402856 29909 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:14.405267 28895 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 3 to 4, leader changed from 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 4 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:14.596740 29849 raft_consensus.cc:979] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791: : Instructing follower 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:14.596916 29849 raft_consensus.cc:1067] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Signalling peer 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:14.597505 29157 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:34474
I20240502 21:27:14.597718 29157 raft_consensus.cc:482] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:14.597827 29157 raft_consensus.cc:3037] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:14.599923 29157 raft_consensus.cc:504] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:14.600688 29157 leader_election.cc:288] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 3 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297)
I20240502 21:27:14.600970 29081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:14.601075 29476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "501feae33ef0484f956ca7a1e090ae01"
I20240502 21:27:14.601176 29081 raft_consensus.cc:3032] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:14.601274 29081 raft_consensus.cc:729] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:14.601284 29476 raft_consensus.cc:3037] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:14.601519 29081 consensus_queue.cc:260] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:14.601799 29081 raft_consensus.cc:3037] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:14.603058 29476 raft_consensus.cc:2454] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 3.
I20240502 21:27:14.603420 29122 leader_election.cc:302] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 501feae33ef0484f956ca7a1e090ae01, 5f1d764da7eb4f0c9fd4cabdca685d7b; no voters:
I20240502 21:27:14.603933 29901 raft_consensus.cc:2781] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:14.603935 29081 raft_consensus.cc:2454] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 3.
I20240502 21:27:14.604081 29901 raft_consensus.cc:686] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:14.604321 29901 consensus_queue.cc:237] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:14.606696 28895 catalog_manager.cc:5484] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 2 to 3, leader changed from 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 3 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:14.614166 29871 tablet_copy_client.cc:523] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:14.617055 29871 tablet_bootstrap.cc:492] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01: Bootstrap starting.
I20240502 21:27:14.637398 29876 tablet_copy_client.cc:523] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:14.639701 29876 tablet_bootstrap.cc:492] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23: Bootstrap starting.
I20240502 21:27:14.645622 29880 tablet_copy_client.cc:523] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:14.647152 29871 tablet_bootstrap.cc:492] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:14.647434 29871 tablet_bootstrap.cc:492] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01: Bootstrap complete.
I20240502 21:27:14.647646 29871 ts_tablet_manager.cc:1367] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.000s
I20240502 21:27:14.648190 29871 raft_consensus.cc:348] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:14.648432 29871 raft_consensus.cc:729] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 501feae33ef0484f956ca7a1e090ae01, State: Initialized, Role: LEARNER
I20240502 21:27:14.648680 29880 tablet_bootstrap.cc:492] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap starting.
I20240502 21:27:14.648648 29871 consensus_queue.cc:260] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:14.649298 29871 ts_tablet_manager.cc:1398] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 21:27:14.650709 29167 tablet_copy_service.cc:342] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Request end of tablet copy session 501feae33ef0484f956ca7a1e090ae01-5010a09a5243452dad7cd9b90535a229 received from {username='slave'} at 127.0.0.1:42866
I20240502 21:27:14.650843 29167 tablet_copy_service.cc:433] P 5f1d764da7eb4f0c9fd4cabdca685d7b: ending tablet copy session 501feae33ef0484f956ca7a1e090ae01-5010a09a5243452dad7cd9b90535a229 on tablet 5010a09a5243452dad7cd9b90535a229 with peer 501feae33ef0484f956ca7a1e090ae01
I20240502 21:27:14.667222 29849 raft_consensus.cc:979] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791: : Instructing follower 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:14.667371 29916 raft_consensus.cc:1067] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Signalling peer 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:14.668049 29150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:34474
I20240502 21:27:14.668236 29150 raft_consensus.cc:482] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:14.668445 29150 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:14.670588 29150 raft_consensus.cc:504] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:14.671545 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:14.671758 29001 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:14.672019 29150 leader_election.cc:288] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 3 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:14.672101 29081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:14.672305 29081 raft_consensus.cc:3032] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:14.672408 29081 raft_consensus.cc:729] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:14.672678 29081 consensus_queue.cc:260] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:14.673147 29081 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:14.673799 29001 raft_consensus.cc:2454] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 3.
I20240502 21:27:14.674147 29121 leader_election.cc:302] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:27:14.674403 29901 raft_consensus.cc:2781] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:14.674552 29901 raft_consensus.cc:686] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:14.674813 29901 consensus_queue.cc:237] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:14.675935 29081 raft_consensus.cc:2454] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 3.
I20240502 21:27:14.677613 28895 catalog_manager.cc:5484] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 2 to 3, leader changed from 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 3 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:14.678769 29876 tablet_bootstrap.cc:492] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:14.679126 29876 tablet_bootstrap.cc:492] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23: Bootstrap complete.
I20240502 21:27:14.680652 29876 ts_tablet_manager.cc:1367] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23: Time spent bootstrapping tablet: real 0.040s user 0.032s sys 0.004s
I20240502 21:27:14.681627 29876 raft_consensus.cc:348] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:14.681866 29876 raft_consensus.cc:729] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9c60776a34a54a09822291687d162a23, State: Initialized, Role: LEARNER
I20240502 21:27:14.682077 29876 consensus_queue.cc:260] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:14.682673 29876 ts_tablet_manager.cc:1398] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:14.683255 29167 tablet_copy_service.cc:342] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Request end of tablet copy session 9c60776a34a54a09822291687d162a23-7ce57dd3a92645839765ba1ce6609430 received from {username='slave'} at 127.0.0.1:42872
I20240502 21:27:14.683383 29167 tablet_copy_service.cc:433] P 5f1d764da7eb4f0c9fd4cabdca685d7b: ending tablet copy session 9c60776a34a54a09822291687d162a23-7ce57dd3a92645839765ba1ce6609430 on tablet 7ce57dd3a92645839765ba1ce6609430 with peer 9c60776a34a54a09822291687d162a23
I20240502 21:27:14.685848 29150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34466
I20240502 21:27:14.686094 29150 raft_consensus.cc:595] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Received request to transfer leadership
I20240502 21:27:14.688964 29889 tablet_copy_client.cc:523] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:14.691856 29150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34466
I20240502 21:27:14.692014 29150 raft_consensus.cc:595] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Received request to transfer leadership
I20240502 21:27:14.692699 29889 tablet_bootstrap.cc:492] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap starting.
I20240502 21:27:14.693172 29150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34466
I20240502 21:27:14.693331 29150 raft_consensus.cc:595] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Received request to transfer leadership
W20240502 21:27:14.693774 28953 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 6d08f3643db34c8ba4d4391b630be366 already in progress
W20240502 21:27:14.693859 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 6d08f3643db34c8ba4d4391b630be366 already in progress
I20240502 21:27:14.710424 29880 tablet_bootstrap.cc:492] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=85 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:14.710793 29880 tablet_bootstrap.cc:492] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap complete.
I20240502 21:27:14.711084 29880 ts_tablet_manager.cc:1367] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent bootstrapping tablet: real 0.062s user 0.035s sys 0.012s
I20240502 21:27:14.711843 29880 raft_consensus.cc:348] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:14.712109 29880 raft_consensus.cc:729] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: c0c2fc14113d41c498fd6abcbbbd03c7, State: Initialized, Role: LEARNER
I20240502 21:27:14.712344 29880 consensus_queue.cc:260] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:14.712908 29880 ts_tablet_manager.cc:1398] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:27:14.713485 29017 tablet_copy_service.cc:342] P 94f26f2c271544729391496f0fa81a58: Request end of tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-54a27d02080d420382add870508aaad4 received from {username='slave'} at 127.0.0.1:54710
I20240502 21:27:14.713618 29017 tablet_copy_service.cc:433] P 94f26f2c271544729391496f0fa81a58: ending tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-54a27d02080d420382add870508aaad4 on tablet 54a27d02080d420382add870508aaad4 with peer c0c2fc14113d41c498fd6abcbbbd03c7
I20240502 21:27:14.728593 29889 tablet_bootstrap.cc:492] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=112 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:14.728909 29889 tablet_bootstrap.cc:492] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap complete.
I20240502 21:27:14.729197 29889 ts_tablet_manager.cc:1367] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent bootstrapping tablet: real 0.037s user 0.031s sys 0.003s
I20240502 21:27:14.729892 29889 raft_consensus.cc:348] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:14.730139 29889 raft_consensus.cc:729] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c0c2fc14113d41c498fd6abcbbbd03c7, State: Initialized, Role: LEARNER
I20240502 21:27:14.730345 29889 consensus_queue.cc:260] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:14.730875 29889 ts_tablet_manager.cc:1398] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:14.731567 29093 tablet_copy_service.cc:342] P 9846ea932cb14dff98f20d67f1829791: Request end of tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-6d08f3643db34c8ba4d4391b630be366 received from {username='slave'} at 127.0.0.1:60738
I20240502 21:27:14.731699 29093 tablet_copy_service.cc:433] P 9846ea932cb14dff98f20d67f1829791: ending tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-6d08f3643db34c8ba4d4391b630be366 on tablet 6d08f3643db34c8ba4d4391b630be366 with peer c0c2fc14113d41c498fd6abcbbbd03c7
I20240502 21:27:14.742436 29894 tablet_copy_client.cc:523] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:14.745276 29894 tablet_bootstrap.cc:492] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap starting.
I20240502 21:27:14.780176 29894 tablet_bootstrap.cc:492] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=94 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:14.780561 29894 tablet_bootstrap.cc:492] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap complete.
I20240502 21:27:14.780831 29894 ts_tablet_manager.cc:1367] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent bootstrapping tablet: real 0.036s user 0.036s sys 0.000s
I20240502 21:27:14.781551 29894 raft_consensus.cc:348] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:14.781769 29894 raft_consensus.cc:729] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: c0c2fc14113d41c498fd6abcbbbd03c7, State: Initialized, Role: LEARNER
I20240502 21:27:14.781976 29894 consensus_queue.cc:260] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:14.782560 29894 ts_tablet_manager.cc:1398] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:14.783277 29488 tablet_copy_service.cc:342] P 501feae33ef0484f956ca7a1e090ae01: Request end of tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-af41f02157c94f55b2bc5090f566592e received from {username='slave'} at 127.0.0.1:60412
I20240502 21:27:14.783394 29488 tablet_copy_service.cc:433] P 501feae33ef0484f956ca7a1e090ae01: ending tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-af41f02157c94f55b2bc5090f566592e on tablet af41f02157c94f55b2bc5090f566592e with peer c0c2fc14113d41c498fd6abcbbbd03c7
I20240502 21:27:14.837739 29477 raft_consensus.cc:3037] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Advancing to term 3
I20240502 21:27:14.861855 29081 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 21:27:14.862382 29909 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:14.866144 29150 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
W20240502 21:27:14.866371 28971 consensus_peers.cc:485] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 -> Peer 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791): Couldn't send request to peer 9c60776a34a54a09822291687d162a23. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:14.867357 29909 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:14.971068 29402 raft_consensus.cc:1203] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.30->[3.31-3.31] Dedup: 3.31->[]
I20240502 21:27:15.065449 29552 raft_consensus.cc:1203] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240502 21:27:15.079692 29476 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:15.080193 29901 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:15.083350 29076 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:15.083796 29933 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:15.107031 29001 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240502 21:27:15.107419 29933 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:15.109573 29901 raft_consensus.cc:979] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b: : Instructing follower 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:15.109766 29933 raft_consensus.cc:1067] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Signalling peer 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:15.110291 29001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:52250
I20240502 21:27:15.111047 29001 raft_consensus.cc:482] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:15.111151 29001 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:15.113402 29001 raft_consensus.cc:504] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:15.114030 29001 leader_election.cc:288] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 4 election: Requested vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:15.114463 29076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:15.114502 29150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:15.114720 29150 raft_consensus.cc:3032] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:15.114604 29076 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:15.114787 29150 raft_consensus.cc:729] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:15.115031 29150 consensus_queue.cc:260] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:15.115386 29150 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:15.116606 29076 raft_consensus.cc:2454] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 4.
I20240502 21:27:15.117272 28972 leader_election.cc:302] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:15.117499 29947 raft_consensus.cc:2781] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:15.117599 29947 raft_consensus.cc:686] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:15.117803 29947 consensus_queue.cc:237] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:15.118145 29150 raft_consensus.cc:2454] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 4.
I20240502 21:27:15.120038 28895 catalog_manager.cc:5484] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 3 to 4, leader changed from 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 4 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:15.148214 29402 raft_consensus.cc:1203] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.31->[4.32-4.32] Dedup: 4.32->[]
I20240502 21:27:15.405026 29916 raft_consensus.cc:1050] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791: attempting to promote NON_VOTER 501feae33ef0484f956ca7a1e090ae01 to VOTER
I20240502 21:27:15.405754 29916 consensus_queue.cc:237] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:15.408135 29476 raft_consensus.cc:1261] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [term 3 LEARNER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:15.408569 29001 raft_consensus.cc:1261] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:15.408974 29150 raft_consensus.cc:1261] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:15.409072 29958 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:15.409686 29916 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:15.409981 29954 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:15.413479 29916 raft_consensus.cc:2932] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:15.415047 29001 raft_consensus.cc:2932] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:15.416216 29947 raft_consensus.cc:1050] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58: attempting to promote NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 to VOTER
I20240502 21:27:15.416859 29947 consensus_queue.cc:237] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 26, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:15.416776 29150 raft_consensus.cc:2932] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:15.420647 29476 raft_consensus.cc:2932] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:15.421047 28903 catalog_manager.cc:5484] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index 27 to 29, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:15.425797 29076 consensus_queue.cc:237] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:15.425968 29081 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 21:27:15.426026 29402 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 LEARNER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 21:27:15.426537 29926 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:15.427016 29926 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:15.427839 29552 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240502 21:27:15.428272 29926 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:15.428491 29150 raft_consensus.cc:1261] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 21:27:15.429981 29916 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:15.431164 29476 raft_consensus.cc:1261] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240502 21:27:15.431638 29954 consensus_queue.cc:1035] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:15.431478 29926 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.432305 29958 raft_consensus.cc:2932] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:15.433847 29076 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.434568 29477 raft_consensus.cc:2932] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:15.435495 28890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5010a09a5243452dad7cd9b90535a229 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:15.436434 29402 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.438201 29150 raft_consensus.cc:2932] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:15.438653 28902 catalog_manager.cc:5484] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 reported cstate change: config changed from index 29 to 30, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New cstate: current_term: 3 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:15.441421 28903 catalog_manager.cc:5484] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 reported cstate change: config changed from index 31 to 32, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.442102 28994 tablet_service.cc:1514] Processing DeleteTablet for tablet 5010a09a5243452dad7cd9b90535a229 with delete_type TABLET_DATA_TOMBSTONED (TS 94f26f2c271544729391496f0fa81a58 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:52236
I20240502 21:27:15.446350 29552 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.446959 29965 tablet_replica.cc:330] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58: stopping tablet replica
I20240502 21:27:15.447147 29965 raft_consensus.cc:2229] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:15.447404 29965 raft_consensus.cc:2258] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:15.448364 29965 ts_tablet_manager.cc:1875] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:15.451520 29003 consensus_queue.cc:237] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:15.453119 29552 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 21:27:15.453140 29402 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240502 21:27:15.453542 29947 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 21:27:15.454514 29965 ts_tablet_manager.cc:1888] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20240502 21:27:15.454627 29965 log.cc:1196] T 5010a09a5243452dad7cd9b90535a229 P 94f26f2c271544729391496f0fa81a58: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-0-root/wals/5010a09a5243452dad7cd9b90535a229
I20240502 21:27:15.454818 29966 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 21:27:15.455240 28889 catalog_manager.cc:4830] TS 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): tablet 5010a09a5243452dad7cd9b90535a229 (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:15.455782 29947 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.457607 29402 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.459650 28889 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54a27d02080d420382add870508aaad4 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:15.459836 29552 raft_consensus.cc:2932] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.460111 28903 catalog_manager.cc:5484] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index 32 to 33, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New cstate: current_term: 3 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:15.464843 29071 tablet_service.cc:1514] Processing DeleteTablet for tablet 54a27d02080d420382add870508aaad4 with delete_type TABLET_DATA_TOMBSTONED (TS 9846ea932cb14dff98f20d67f1829791 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:51584
I20240502 21:27:15.465598 29967 tablet_replica.cc:330] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791: stopping tablet replica
I20240502 21:27:15.465899 29967 raft_consensus.cc:2229] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:15.466112 29967 raft_consensus.cc:2258] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:15.466908 29967 ts_tablet_manager.cc:1875] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:15.472303 29967 ts_tablet_manager.cc:1888] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20240502 21:27:15.472440 29967 log.cc:1196] T 54a27d02080d420382add870508aaad4 P 9846ea932cb14dff98f20d67f1829791: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-1-root/wals/54a27d02080d420382add870508aaad4
I20240502 21:27:15.473042 28890 catalog_manager.cc:4830] TS 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): tablet 54a27d02080d420382add870508aaad4 (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:15.497076 29076 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 21:27:15.497511 29966 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:15.505847 29402 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEARNER]: Advancing to term 4
I20240502 21:27:15.505923 29157 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 21:27:15.506676 29966 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:15.508708 29402 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 LEARNER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240502 21:27:15.509179 29966 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:15.514307 29966 raft_consensus.cc:1050] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58: attempting to promote NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 to VOTER
I20240502 21:27:15.515092 29966 consensus_queue.cc:237] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 28, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:15.516846 29157 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 21:27:15.516978 29402 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 LEARNER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 21:27:15.517328 29081 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 21:27:15.517277 29955 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:15.517740 29955 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:15.517932 29926 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:15.519950 29926 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Committing config change with OpId 4.30: config changed from index 27 to 30, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.520488 29402 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 27 to 30, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.520877 29081 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 27 to 30, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.525012 28902 catalog_manager.cc:5484] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index 27 to 30, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:15.524873 29157 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 27 to 30, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.529279 29003 consensus_queue.cc:237] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:15.531061 29081 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 21:27:15.531241 29402 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240502 21:27:15.531661 29955 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:15.531919 29966 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:15.533951 29947 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.534931 29402 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.535243 29081 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.537395 28889 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d08f3643db34c8ba4d4391b630be366 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:15.538381 28895 catalog_manager.cc:5484] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index 30 to 31, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New cstate: current_term: 4 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:15.542402 29138 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d08f3643db34c8ba4d4391b630be366 with delete_type TABLET_DATA_TOMBSTONED (TS 5f1d764da7eb4f0c9fd4cabdca685d7b not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:34466
I20240502 21:27:15.543159 29971 tablet_replica.cc:330] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b: stopping tablet replica
I20240502 21:27:15.543371 29971 raft_consensus.cc:2229] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:15.543633 29971 raft_consensus.cc:2258] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:15.544380 29971 ts_tablet_manager.cc:1875] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:15.549813 29971 ts_tablet_manager.cc:1888] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.30
I20240502 21:27:15.549932 29971 log.cc:1196] T 6d08f3643db34c8ba4d4391b630be366 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-2-root/wals/6d08f3643db34c8ba4d4391b630be366
I20240502 21:27:15.550575 28890 catalog_manager.cc:4830] TS 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): tablet 6d08f3643db34c8ba4d4391b630be366 (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:15.624855 29901 raft_consensus.cc:1050] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b: attempting to promote NON_VOTER 9c60776a34a54a09822291687d162a23 to VOTER
I20240502 21:27:15.625659 29901 consensus_queue.cc:237] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:15.627643 29552 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 1 LEARNER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 21:27:15.627671 29081 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 21:27:15.628108 29003 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 21:27:15.628167 29972 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:15.628495 29953 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:15.629004 29972 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 21:27:15.635661 29972 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:15.639130 29003 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:15.640214 29552 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:15.641669 29081 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:15.645809 28902 catalog_manager.cc:5484] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index 25 to 26, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:15.650341 29157 consensus_queue.cc:237] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:15.651770 29003 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 21:27:15.651851 29552 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 1 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240502 21:27:15.652295 29953 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:15.652594 29901 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:15.654760 29953 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:15.655346 29552 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:15.656157 29003 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:15.659147 28895 catalog_manager.cc:5484] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 reported cstate change: config changed from index 26 to 27, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New cstate: current_term: 1 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:15.660387 28890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7ce57dd3a92645839765ba1ce6609430 with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:15.662540 29071 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ce57dd3a92645839765ba1ce6609430 with delete_type TABLET_DATA_TOMBSTONED (TS 9846ea932cb14dff98f20d67f1829791 not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:51584
I20240502 21:27:15.662889 29967 tablet_replica.cc:330] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791: stopping tablet replica
I20240502 21:27:15.663095 29967 raft_consensus.cc:2229] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:15.663302 29967 raft_consensus.cc:2258] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:15.664088 29967 ts_tablet_manager.cc:1875] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:15.669423 29967 ts_tablet_manager.cc:1888] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20240502 21:27:15.669534 29967 log.cc:1196] T 7ce57dd3a92645839765ba1ce6609430 P 9846ea932cb14dff98f20d67f1829791: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-1-root/wals/7ce57dd3a92645839765ba1ce6609430
I20240502 21:27:15.670171 28890 catalog_manager.cc:4830] TS 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): tablet 7ce57dd3a92645839765ba1ce6609430 (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:15.719075 29979 raft_consensus.cc:1050] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: attempting to promote NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 to VOTER
I20240502 21:27:15.720077 29979 consensus_queue.cc:237] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 30, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:15.721946 29150 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Refusing update from remote peer 501feae33ef0484f956ca7a1e090ae01: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 21:27:15.721967 29402 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 LEARNER]: Refusing update from remote peer 501feae33ef0484f956ca7a1e090ae01: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 21:27:15.722354 29003 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Refusing update from remote peer 501feae33ef0484f956ca7a1e090ae01: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 21:27:15.722579 29979 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 21:27:15.722944 29919 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 21:27:15.723248 29980 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 21:27:15.729019 29919 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.732638 29003 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.732977 28903 catalog_manager.cc:5484] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 reported cstate change: config changed from index 32 to 33, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "501feae33ef0484f956ca7a1e090ae01" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:15.735749 29402 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.736785 29476 consensus_queue.cc:237] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:15.738638 29401 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Refusing update from remote peer 501feae33ef0484f956ca7a1e090ae01: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240502 21:27:15.739115 29003 raft_consensus.cc:1261] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Refusing update from remote peer 501feae33ef0484f956ca7a1e090ae01: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240502 21:27:15.739868 29919 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 21:27:15.739609 29150 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.741923 29919 consensus_queue.cc:1035] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 21:27:15.745036 29401 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.744467 29980 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.748471 28903 catalog_manager.cc:5484] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7 reported cstate change: config changed from index 33 to 34, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New cstate: current_term: 4 leader_uuid: "501feae33ef0484f956ca7a1e090ae01" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.750267 28951 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af41f02157c94f55b2bc5090f566592e with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:15.751271 29138 tablet_service.cc:1514] Processing DeleteTablet for tablet af41f02157c94f55b2bc5090f566592e with delete_type TABLET_DATA_TOMBSTONED (TS 5f1d764da7eb4f0c9fd4cabdca685d7b not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:34466
I20240502 21:27:15.751579 29971 tablet_replica.cc:330] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b: stopping tablet replica
I20240502 21:27:15.751796 29971 raft_consensus.cc:2229] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:15.751901 29003 raft_consensus.cc:2932] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:15.752192 29971 raft_consensus.cc:2258] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:15.753320 29971 ts_tablet_manager.cc:1875] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:15.758347 29971 ts_tablet_manager.cc:1888] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.33
I20240502 21:27:15.758442 29971 log.cc:1196] T af41f02157c94f55b2bc5090f566592e P 5f1d764da7eb4f0c9fd4cabdca685d7b: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-2-root/wals/af41f02157c94f55b2bc5090f566592e
I20240502 21:27:15.759003 28890 catalog_manager.cc:4830] TS 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): tablet af41f02157c94f55b2bc5090f566592e (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:15.970131 29920 tablet_copy_client.cc:523] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:15.972844 29920 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23: Bootstrap starting.
I20240502 21:27:16.012833 29920 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:16.013149 29920 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23: Bootstrap complete.
I20240502 21:27:16.013371 29920 ts_tablet_manager.cc:1367] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23: Time spent bootstrapping tablet: real 0.041s user 0.033s sys 0.006s
I20240502 21:27:16.014015 29920 raft_consensus.cc:348] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:16.014200 29920 raft_consensus.cc:729] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 9c60776a34a54a09822291687d162a23, State: Initialized, Role: LEARNER
I20240502 21:27:16.014384 29920 consensus_queue.cc:260] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:16.014807 29920 ts_tablet_manager.cc:1398] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:16.015609 29093 tablet_copy_service.cc:342] P 9846ea932cb14dff98f20d67f1829791: Request end of tablet copy session 9c60776a34a54a09822291687d162a23-ea5a9bdae3f3459f915eaf3bdd3d346c received from {username='slave'} at 127.0.0.1:60728
I20240502 21:27:16.015741 29093 tablet_copy_service.cc:433] P 9846ea932cb14dff98f20d67f1829791: ending tablet copy session 9c60776a34a54a09822291687d162a23-ea5a9bdae3f3459f915eaf3bdd3d346c on tablet ea5a9bdae3f3459f915eaf3bdd3d346c with peer 9c60776a34a54a09822291687d162a23
I20240502 21:27:16.279047 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:16.281927 29150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
new_leader_uuid: "9846ea932cb14dff98f20d67f1829791"
from {username='slave'} at 127.0.0.1:42850
I20240502 21:27:16.282174 29150 raft_consensus.cc:595] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Received request to transfer leadership to TS 9846ea932cb14dff98f20d67f1829791
I20240502 21:27:16.283190 29003 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
new_leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:52268
I20240502 21:27:16.283332 29003 raft_consensus.cc:595] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Received request to transfer leadership to TS 5f1d764da7eb4f0c9fd4cabdca685d7b
I20240502 21:27:16.284092 29003 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ee4eba5acb8d4af986a00f937de0270e"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
new_leader_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7"
from {username='slave'} at 127.0.0.1:52268
I20240502 21:27:16.284212 29003 raft_consensus.cc:595] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Received request to transfer leadership to TS c0c2fc14113d41c498fd6abcbbbd03c7
I20240502 21:27:16.284541 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240502 21:27:16.284734 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:16.427776 29552 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 3 LEARNER]: Advancing to term 4
I20240502 21:27:16.490444 29955 raft_consensus.cc:979] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58: : Instructing follower 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:16.490568 29955 raft_consensus.cc:1067] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Signalling peer 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:16.491080 29150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:34468
I20240502 21:27:16.491250 29150 raft_consensus.cc:482] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:16.491317 29150 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:16.493181 29150 raft_consensus.cc:504] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:16.493721 29150 leader_election.cc:288] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 5 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:16.494016 29003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:16.494098 29081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5a9bdae3f3459f915eaf3bdd3d346c" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:16.494194 29003 raft_consensus.cc:3032] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:27:16.494210 29081 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:16.494266 29003 raft_consensus.cc:729] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:16.494505 29003 consensus_queue.cc:260] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:16.494927 29003 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:16.496891 29081 raft_consensus.cc:2454] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 5.
I20240502 21:27:16.497256 29122 leader_election.cc:302] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:16.497433 29003 raft_consensus.cc:2454] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 5.
I20240502 21:27:16.497558 29901 raft_consensus.cc:2781] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:16.497895 29901 raft_consensus.cc:686] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:16.498137 29901 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:16.500268 28904 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 4 to 5, leader changed from 94f26f2c271544729391496f0fa81a58 (127.25.16.65) to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 5 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:16.518535 29990 raft_consensus.cc:979] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58: : Instructing follower c0c2fc14113d41c498fd6abcbbbd03c7 to start an election
I20240502 21:27:16.518638 29990 raft_consensus.cc:1067] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Signalling peer c0c2fc14113d41c498fd6abcbbbd03c7 to start an election
I20240502 21:27:16.519147 29401 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ee4eba5acb8d4af986a00f937de0270e"
dest_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7"
from {username='slave'} at 127.0.0.1:59384
I20240502 21:27:16.519287 29401 raft_consensus.cc:482] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:16.519358 29401 raft_consensus.cc:3037] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:16.521121 29401 raft_consensus.cc:504] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:16.521673 29401 leader_election.cc:288] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [CANDIDATE]: Term 3 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:16.521965 29003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4eba5acb8d4af986a00f937de0270e" candidate_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:16.522034 29081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee4eba5acb8d4af986a00f937de0270e" candidate_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:16.522130 29003 raft_consensus.cc:3032] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:16.522146 29081 raft_consensus.cc:3037] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:16.522204 29003 raft_consensus.cc:729] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:16.522420 29003 consensus_queue.cc:260] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:16.522770 29003 raft_consensus.cc:3037] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:16.524708 29081 raft_consensus.cc:2454] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0c2fc14113d41c498fd6abcbbbd03c7 in term 3.
I20240502 21:27:16.525043 29367 leader_election.cc:302] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9846ea932cb14dff98f20d67f1829791, c0c2fc14113d41c498fd6abcbbbd03c7; no voters:
I20240502 21:27:16.525275 29003 raft_consensus.cc:2454] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0c2fc14113d41c498fd6abcbbbd03c7 in term 3.
I20240502 21:27:16.525295 29987 raft_consensus.cc:2781] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:16.525624 29987 raft_consensus.cc:686] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 LEADER]: Becoming Leader. State: Replica: c0c2fc14113d41c498fd6abcbbbd03c7, State: Running, Role: LEADER
I20240502 21:27:16.525866 29987 consensus_queue.cc:237] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:16.528003 28904 catalog_manager.cc:5484] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 reported cstate change: term changed from 2 to 3, leader changed from 94f26f2c271544729391496f0fa81a58 (127.25.16.65) to c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68). New cstate: current_term: 3 leader_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:16.720644 29991 raft_consensus.cc:979] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b: : Instructing follower 9846ea932cb14dff98f20d67f1829791 to start an election
I20240502 21:27:16.720798 29991 raft_consensus.cc:1067] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Signalling peer 9846ea932cb14dff98f20d67f1829791 to start an election
I20240502 21:27:16.721350 29081 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
from {username='slave'} at 127.0.0.1:51598
I20240502 21:27:16.721509 29081 raft_consensus.cc:482] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:16.721577 29081 raft_consensus.cc:3037] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:16.723435 29081 raft_consensus.cc:504] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:16.723956 29081 leader_election.cc:288] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 4 election: Requested vote from peers 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785), 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297)
I20240502 21:27:16.724321 29150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:16.724509 29150 raft_consensus.cc:3032] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:16.724505 29476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab35657135e44445aee4aa6961f70c27" candidate_uuid: "9846ea932cb14dff98f20d67f1829791" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "501feae33ef0484f956ca7a1e090ae01"
I20240502 21:27:16.724602 29150 raft_consensus.cc:729] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:16.724709 29476 raft_consensus.cc:3037] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:16.724862 29150 consensus_queue.cc:260] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:16.725211 29150 raft_consensus.cc:3037] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:16.727384 29476 raft_consensus.cc:2454] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 4.
I20240502 21:27:16.727627 29150 raft_consensus.cc:2454] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846ea932cb14dff98f20d67f1829791 in term 4.
I20240502 21:27:16.727746 29048 leader_election.cc:302] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 501feae33ef0484f956ca7a1e090ae01, 9846ea932cb14dff98f20d67f1829791; no voters:
I20240502 21:27:16.727989 29954 raft_consensus.cc:2781] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:16.728327 29954 raft_consensus.cc:686] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 4 LEADER]: Becoming Leader. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:16.728562 29954 consensus_queue.cc:237] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:16.730743 28904 catalog_manager.cc:5484] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: term changed from 3 to 4, leader changed from 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) to 9846ea932cb14dff98f20d67f1829791 (127.25.16.66). New cstate: current_term: 4 leader_uuid: "9846ea932cb14dff98f20d67f1829791" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:16.761395 29401 consensus_queue.cc:237] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:16.763502 29150 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer c0c2fc14113d41c498fd6abcbbbd03c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 21:27:16.764206 29987 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:16.764358 29081 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer c0c2fc14113d41c498fd6abcbbbd03c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 21:27:16.765163 29987 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:16.767319 30003 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:16.767910 29150 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:16.768831 29081 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:16.772657 28903 catalog_manager.cc:5484] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 reported cstate change: config changed from index 28 to 30, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New cstate: current_term: 2 leader_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:16.772958 29003 consensus_queue.cc:237] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
W20240502 21:27:16.775579 29366 consensus_peers.cc:485] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 -> Peer 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791): Couldn't send request to peer 9c60776a34a54a09822291687d162a23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccd9ffc92c5c47d088c530e919f2ff77. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:16.777056 28972 consensus_peers.cc:485] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 -> Peer 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297): Couldn't send request to peer 501feae33ef0484f956ca7a1e090ae01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d08f3643db34c8ba4d4391b630be366. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:16.777141 29401 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 21:27:16.777638 29081 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 21:27:16.777700 29990 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:16.778177 29990 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:16.779944 29990 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 31 to 32, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:16.780503 29401 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:16.783437 29081 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:16.784174 29150 consensus_queue.cc:237] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:16.786162 29476 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 21:27:16.786325 29003 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240502 21:27:16.788061 28904 catalog_manager.cc:5484] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 reported cstate change: config changed from index 31 to 32, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New cstate: current_term: 4 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
W20240502 21:27:16.788336 29121 consensus_peers.cc:485] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b -> Peer 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791): Couldn't send request to peer 9c60776a34a54a09822291687d162a23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31c63a7053fa45b69eba6d7f7e78139b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:16.786688 29991 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:16.788861 29901 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:16.792070 30014 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:16.792680 29003 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:16.793004 29476 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } } }
I20240502 21:27:16.796834 28895 catalog_manager.cc:5484] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index 30 to 31, NON_VOTER 9c60776a34a54a09822291687d162a23 (127.25.16.70) added. New cstate: current_term: 3 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:16.798038 28953 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240502 21:27:16.807047 28953 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ab35657135e44445aee4aa6961f70c27, TS 5f1d764da7eb4f0c9fd4cabdca685d7b -> TS 9c60776a34a54a09822291687d162a23 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
W20240502 21:27:16.807149 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ab35657135e44445aee4aa6961f70c27, TS 5f1d764da7eb4f0c9fd4cabdca685d7b -> TS 9c60776a34a54a09822291687d162a23 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
W20240502 21:27:16.815764 28953 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1de7fda467344968ab81f653b8501210, TS 9846ea932cb14dff98f20d67f1829791 -> TS 501feae33ef0484f956ca7a1e090ae01 move failed, destination replica disappeared from tablet's Raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
W20240502 21:27:16.815842 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 1de7fda467344968ab81f653b8501210, TS 9846ea932cb14dff98f20d67f1829791 -> TS 501feae33ef0484f956ca7a1e090ae01 move failed, destination replica disappeared from tablet's Raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
W20240502 21:27:16.819622 28953 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ea5a9bdae3f3459f915eaf3bdd3d346c, TS 94f26f2c271544729391496f0fa81a58 -> TS c0c2fc14113d41c498fd6abcbbbd03c7 move failed, destination replica disappeared from tablet's Raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
W20240502 21:27:16.819698 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ea5a9bdae3f3459f915eaf3bdd3d346c, TS 94f26f2c271544729391496f0fa81a58 -> TS c0c2fc14113d41c498fd6abcbbbd03c7 move failed, destination replica disappeared from tablet's Raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:16.887671 29003 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20240502 21:27:16.888134 30014 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:16.891541 29552 raft_consensus.cc:3037] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 4 LEARNER]: Advancing to term 5
I20240502 21:27:16.891738 29081 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20240502 21:27:16.894492 29552 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 LEARNER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20240502 21:27:16.894912 30019 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:16.895159 30014 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:16.898303 30019 raft_consensus.cc:1050] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b: attempting to promote NON_VOTER 9c60776a34a54a09822291687d162a23 to VOTER
I20240502 21:27:16.898993 30019 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 29, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:16.900801 29552 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 LEARNER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240502 21:27:16.901194 29081 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240502 21:27:16.901677 29901 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:16.902139 30019 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:16.903344 29003 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240502 21:27:16.904006 30019 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:16.904719 29901 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Committing config change with OpId 5.31: config changed from index 26 to 31, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:16.905153 29552 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index 26 to 31, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:16.906556 29003 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index 26 to 31, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:16.909511 28903 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 reported cstate change: config changed from index 26 to 31, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:16.909740 29081 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index 26 to 31, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:17.037281 29081 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer c0c2fc14113d41c498fd6abcbbbd03c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:17.037786 30003 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:17.041256 29003 raft_consensus.cc:1261] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Refusing update from remote peer c0c2fc14113d41c498fd6abcbbbd03c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240502 21:27:17.042680 29987 consensus_queue.cc:1035] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:17.218492 29476 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 21:27:17.219038 29954 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:17.222397 29150 raft_consensus.cc:1261] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Refusing update from remote peer 9846ea932cb14dff98f20d67f1829791: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240502 21:27:17.222913 29954 consensus_queue.cc:1035] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:17.290642 30029 ts_tablet_manager.cc:902] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23: Initiating tablet copy from peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:17.291929 30029 tablet_copy_client.cc:308] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.67:41785
I20240502 21:27:17.292510 29167 tablet_copy_service.cc:140] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Received BeginTabletCopySession request for tablet 31c63a7053fa45b69eba6d7f7e78139b from peer 9c60776a34a54a09822291687d162a23 ({username='slave'} at 127.0.0.1:42872)
I20240502 21:27:17.292688 29167 tablet_copy_service.cc:161] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Beginning new tablet copy session on tablet 31c63a7053fa45b69eba6d7f7e78139b from peer 9c60776a34a54a09822291687d162a23 at {username='slave'} at 127.0.0.1:42872: session id = 9c60776a34a54a09822291687d162a23-31c63a7053fa45b69eba6d7f7e78139b
I20240502 21:27:17.294960 29167 tablet_copy_source_session.cc:205] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:17.296032 30029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c63a7053fa45b69eba6d7f7e78139b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.301548 30029 tablet_copy_client.cc:783] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:17.302071 30029 tablet_copy_client.cc:647] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:17.303282 30033 tablet_copy_client.cc:1002] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:17.327872 30034 ts_tablet_manager.cc:902] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23: Initiating tablet copy from peer c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499)
I20240502 21:27:17.329136 30034 tablet_copy_client.cc:308] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.68:45499
I20240502 21:27:17.335052 29412 tablet_copy_service.cc:140] P c0c2fc14113d41c498fd6abcbbbd03c7: Received BeginTabletCopySession request for tablet ccd9ffc92c5c47d088c530e919f2ff77 from peer 9c60776a34a54a09822291687d162a23 ({username='slave'} at 127.0.0.1:51900)
I20240502 21:27:17.335166 29412 tablet_copy_service.cc:161] P c0c2fc14113d41c498fd6abcbbbd03c7: Beginning new tablet copy session on tablet ccd9ffc92c5c47d088c530e919f2ff77 from peer 9c60776a34a54a09822291687d162a23 at {username='slave'} at 127.0.0.1:51900: session id = 9c60776a34a54a09822291687d162a23-ccd9ffc92c5c47d088c530e919f2ff77
I20240502 21:27:17.337419 29412 tablet_copy_source_session.cc:205] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:17.338477 30034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd9ffc92c5c47d088c530e919f2ff77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.343183 30034 tablet_copy_client.cc:783] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:17.343503 30034 tablet_copy_client.cc:647] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:17.344766 30038 tablet_copy_client.cc:1002] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:17.357657 30041 ts_tablet_manager.cc:902] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01: Initiating tablet copy from peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159)
I20240502 21:27:17.358989 30041 tablet_copy_client.cc:308] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.65:33159
I20240502 21:27:17.359526 29017 tablet_copy_service.cc:140] P 94f26f2c271544729391496f0fa81a58: Received BeginTabletCopySession request for tablet 6d08f3643db34c8ba4d4391b630be366 from peer 501feae33ef0484f956ca7a1e090ae01 ({username='slave'} at 127.0.0.1:48068)
I20240502 21:27:17.359637 29017 tablet_copy_service.cc:161] P 94f26f2c271544729391496f0fa81a58: Beginning new tablet copy session on tablet 6d08f3643db34c8ba4d4391b630be366 from peer 501feae33ef0484f956ca7a1e090ae01 at {username='slave'} at 127.0.0.1:48068: session id = 501feae33ef0484f956ca7a1e090ae01-6d08f3643db34c8ba4d4391b630be366
I20240502 21:27:17.361745 29017 tablet_copy_source_session.cc:205] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:17.362685 30041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d08f3643db34c8ba4d4391b630be366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:17.368180 30041 tablet_copy_client.cc:783] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:17.368417 30041 tablet_copy_client.cc:647] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:17.370236 30045 tablet_copy_client.cc:1002] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:18.285015 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:18.287977 29150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
new_leader_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:42850
I20240502 21:27:18.288170 29150 raft_consensus.cc:595] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Received request to transfer leadership to TS 94f26f2c271544729391496f0fa81a58
I20240502 21:27:18.289139 29081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "93477dec2c574c49a0bc9a7f022ad7fa"
dest_uuid: "9846ea932cb14dff98f20d67f1829791"
mode: GRACEFUL
new_leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:60752
I20240502 21:27:18.289299 29081 raft_consensus.cc:595] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Received request to transfer leadership to TS 5f1d764da7eb4f0c9fd4cabdca685d7b
I20240502 21:27:18.289666 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 21:27:18.289896 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:18.537806 30047 raft_consensus.cc:979] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: : Instructing follower 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:18.537976 30047 raft_consensus.cc:1067] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Signalling peer 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:18.538513 29003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:52250
I20240502 21:27:18.538717 29003 raft_consensus.cc:482] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:18.538812 29003 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:18.541427 29003 raft_consensus.cc:504] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:18.542073 29003 leader_election.cc:288] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 4 election: Requested vote from peers 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785), 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297)
I20240502 21:27:18.542299 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:18.542466 29157 raft_consensus.cc:3032] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:18.542520 29476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "501feae33ef0484f956ca7a1e090ae01"
I20240502 21:27:18.542536 29157 raft_consensus.cc:729] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:18.542706 29476 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:18.542883 29157 consensus_queue.cc:260] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:18.543308 29157 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:18.545220 29476 raft_consensus.cc:2454] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 4.
I20240502 21:27:18.545575 28972 leader_election.cc:302] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 501feae33ef0484f956ca7a1e090ae01, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:27:18.545740 29157 raft_consensus.cc:2454] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 4.
I20240502 21:27:18.545833 30050 raft_consensus.cc:2781] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:18.545961 30050 raft_consensus.cc:686] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:18.546197 30050 consensus_queue.cc:237] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:18.548880 28903 catalog_manager.cc:5484] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 3 to 4, leader changed from 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 4 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:18.557499 29003 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52236
I20240502 21:27:18.557632 29003 raft_consensus.cc:595] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Received request to transfer leadership
I20240502 21:27:18.561028 29003 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52236
I20240502 21:27:18.561151 29003 raft_consensus.cc:595] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Received request to transfer leadership
W20240502 21:27:18.561396 28889 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 31c63a7053fa45b69eba6d7f7e78139b already in progress [suppressed 1 similar messages]
I20240502 21:27:18.562152 29003 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52236
I20240502 21:27:18.562261 29003 raft_consensus.cc:595] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Received request to transfer leadership
W20240502 21:27:18.562556 28953 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 31c63a7053fa45b69eba6d7f7e78139b already in progress
W20240502 21:27:18.562618 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 31c63a7053fa45b69eba6d7f7e78139b already in progress
I20240502 21:27:18.680119 30057 raft_consensus.cc:979] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791: : Instructing follower 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:18.680272 30057 raft_consensus.cc:1067] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Signalling peer 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:18.680780 29150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "93477dec2c574c49a0bc9a7f022ad7fa"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:34474
I20240502 21:27:18.680980 29150 raft_consensus.cc:482] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:18.681077 29150 raft_consensus.cc:3037] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:18.682971 29150 raft_consensus.cc:504] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:18.683575 29150 leader_election.cc:288] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 3 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
I20240502 21:27:18.683792 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93477dec2c574c49a0bc9a7f022ad7fa" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:18.683970 29001 raft_consensus.cc:3037] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:18.684196 29081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93477dec2c574c49a0bc9a7f022ad7fa" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "9846ea932cb14dff98f20d67f1829791"
I20240502 21:27:18.684397 29081 raft_consensus.cc:3032] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:18.684466 29081 raft_consensus.cc:729] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9846ea932cb14dff98f20d67f1829791, State: Running, Role: LEADER
I20240502 21:27:18.684677 29081 consensus_queue.cc:260] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:18.685047 29081 raft_consensus.cc:3037] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:18.686373 29001 raft_consensus.cc:2454] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 3.
I20240502 21:27:18.686740 29121 leader_election.cc:302] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:27:18.686887 29081 raft_consensus.cc:2454] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 3.
I20240502 21:27:18.687034 30048 raft_consensus.cc:2781] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:18.687186 30048 raft_consensus.cc:686] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:18.687435 30048 consensus_queue.cc:237] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } }
I20240502 21:27:18.689821 28903 catalog_manager.cc:5484] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 2 to 3, leader changed from 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 3 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:18.988951 29150 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 21:27:18.989452 30050 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:18.991652 30049 raft_consensus.cc:979] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: : Instructing follower 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:18.991863 30050 raft_consensus.cc:1067] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Signalling peer 5f1d764da7eb4f0c9fd4cabdca685d7b to start an election
I20240502 21:27:18.992380 29157 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
from {username='slave'} at 127.0.0.1:34468
I20240502 21:27:19.006793 29476 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240502 21:27:19.007265 30061 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:19.007418 29157 raft_consensus.cc:482] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:19.007545 29157 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Advancing to term 5
W20240502 21:27:19.009073 28971 consensus_peers.cc:485] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 -> Peer 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791): Couldn't send request to peer 9c60776a34a54a09822291687d162a23. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:19.010396 29157 raft_consensus.cc:504] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:19.011564 29157 leader_election.cc:288] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 5 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297)
I20240502 21:27:19.011723 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:19.011818 29477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "501feae33ef0484f956ca7a1e090ae01"
I20240502 21:27:19.011942 29001 raft_consensus.cc:3032] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:27:19.012030 29001 raft_consensus.cc:729] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:19.012310 29001 consensus_queue.cc:260] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:19.012712 29001 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:19.014581 29001 raft_consensus.cc:2454] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1d764da7eb4f0c9fd4cabdca685d7b in term 5.
I20240502 21:27:19.014925 29121 leader_election.cc:302] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters: 501feae33ef0484f956ca7a1e090ae01
I20240502 21:27:19.015206 30048 raft_consensus.cc:2781] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:19.015377 30048 raft_consensus.cc:686] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Becoming Leader. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:19.015655 30048 consensus_queue.cc:237] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:19.017860 28903 catalog_manager.cc:5484] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: term changed from 4 to 5, leader changed from 94f26f2c271544729391496f0fa81a58 (127.25.16.65) to 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67). New cstate: current_term: 5 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:19.125928 29081 raft_consensus.cc:1261] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 21:27:19.126443 30048 consensus_queue.cc:1035] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:19.130645 29003 raft_consensus.cc:1261] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240502 21:27:19.131785 30048 consensus_queue.cc:1035] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240502 21:27:19.303604 30029 tablet_copy_client.cc:523] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:19.307161 30029 tablet_bootstrap.cc:492] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23: Bootstrap starting.
I20240502 21:27:19.345115 30034 tablet_copy_client.cc:523] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:19.348237 30034 tablet_bootstrap.cc:492] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23: Bootstrap starting.
I20240502 21:27:19.367091 30029 tablet_bootstrap.cc:492] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=88 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:19.367460 30029 tablet_bootstrap.cc:492] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23: Bootstrap complete.
I20240502 21:27:19.367731 30029 ts_tablet_manager.cc:1367] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23: Time spent bootstrapping tablet: real 0.061s user 0.060s sys 0.000s
I20240502 21:27:19.368474 30029 raft_consensus.cc:348] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:19.368731 30029 raft_consensus.cc:729] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 9c60776a34a54a09822291687d162a23, State: Initialized, Role: LEARNER
I20240502 21:27:19.368971 30029 consensus_queue.cc:260] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:19.369628 30029 ts_tablet_manager.cc:1398] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 21:27:19.370143 29167 tablet_copy_service.cc:342] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Request end of tablet copy session 9c60776a34a54a09822291687d162a23-31c63a7053fa45b69eba6d7f7e78139b received from {username='slave'} at 127.0.0.1:42872
I20240502 21:27:19.370250 29167 tablet_copy_service.cc:433] P 5f1d764da7eb4f0c9fd4cabdca685d7b: ending tablet copy session 9c60776a34a54a09822291687d162a23-31c63a7053fa45b69eba6d7f7e78139b on tablet 31c63a7053fa45b69eba6d7f7e78139b with peer 9c60776a34a54a09822291687d162a23
I20240502 21:27:19.371279 30041 tablet_copy_client.cc:523] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:19.374599 30041 tablet_bootstrap.cc:492] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01: Bootstrap starting.
I20240502 21:27:19.388134 30034 tablet_bootstrap.cc:492] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:19.388548 30034 tablet_bootstrap.cc:492] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23: Bootstrap complete.
I20240502 21:27:19.388808 30034 ts_tablet_manager.cc:1367] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23: Time spent bootstrapping tablet: real 0.041s user 0.033s sys 0.007s
I20240502 21:27:19.389513 30034 raft_consensus.cc:348] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:19.389714 30034 raft_consensus.cc:729] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9c60776a34a54a09822291687d162a23, State: Initialized, Role: LEARNER
I20240502 21:27:19.389883 30034 consensus_queue.cc:260] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true } }
I20240502 21:27:19.390369 30034 ts_tablet_manager.cc:1398] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:19.391074 29412 tablet_copy_service.cc:342] P c0c2fc14113d41c498fd6abcbbbd03c7: Request end of tablet copy session 9c60776a34a54a09822291687d162a23-ccd9ffc92c5c47d088c530e919f2ff77 received from {username='slave'} at 127.0.0.1:51900
I20240502 21:27:19.391177 29412 tablet_copy_service.cc:433] P c0c2fc14113d41c498fd6abcbbbd03c7: ending tablet copy session 9c60776a34a54a09822291687d162a23-ccd9ffc92c5c47d088c530e919f2ff77 on tablet ccd9ffc92c5c47d088c530e919f2ff77 with peer 9c60776a34a54a09822291687d162a23
I20240502 21:27:19.416111 30041 tablet_bootstrap.cc:492] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=112 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:19.416486 30041 tablet_bootstrap.cc:492] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01: Bootstrap complete.
I20240502 21:27:19.416731 30041 ts_tablet_manager.cc:1367] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01: Time spent bootstrapping tablet: real 0.042s user 0.033s sys 0.009s
I20240502 21:27:19.417364 30041 raft_consensus.cc:348] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:19.417572 30041 raft_consensus.cc:729] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 501feae33ef0484f956ca7a1e090ae01, State: Initialized, Role: LEARNER
I20240502 21:27:19.417755 30041 consensus_queue.cc:260] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:19.418267 30041 ts_tablet_manager.cc:1398] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:19.418813 29017 tablet_copy_service.cc:342] P 94f26f2c271544729391496f0fa81a58: Request end of tablet copy session 501feae33ef0484f956ca7a1e090ae01-6d08f3643db34c8ba4d4391b630be366 received from {username='slave'} at 127.0.0.1:48068
I20240502 21:27:19.418921 29017 tablet_copy_service.cc:433] P 94f26f2c271544729391496f0fa81a58: ending tablet copy session 501feae33ef0484f956ca7a1e090ae01-6d08f3643db34c8ba4d4391b630be366 on tablet 6d08f3643db34c8ba4d4391b630be366 with peer 501feae33ef0484f956ca7a1e090ae01
I20240502 21:27:19.426801 29552 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 3 LEARNER]: Advancing to term 5
I20240502 21:27:19.428896 29552 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 5 LEARNER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240502 21:27:19.429368 30048 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: NON_VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:19.431995 30048 raft_consensus.cc:1050] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: attempting to promote NON_VOTER 9c60776a34a54a09822291687d162a23 to VOTER
W20240502 21:27:19.432102 30048 raft_consensus.cc:1054] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: Unable to promote non-voter 9c60776a34a54a09822291687d162a23: Illegal state: Leader has not yet committed an operation in its own term
I20240502 21:27:19.441002 29552 raft_consensus.cc:1203] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240502 21:27:19.577373 29003 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240502 21:27:19.577888 30047 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 21:27:19.581208 29476 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:19.581708 30047 raft_consensus.cc:1050] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: attempting to promote NON_VOTER 9c60776a34a54a09822291687d162a23 to VOTER
I20240502 21:27:19.582430 30047 consensus_queue.cc:237] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 32, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:19.583721 29476 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240502 21:27:19.584383 29476 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20240502 21:27:19.584865 30075 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 21:27:19.585126 29552 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 5 LEARNER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20240502 21:27:19.585541 30075 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 21:27:19.585654 29003 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20240502 21:27:19.586416 30075 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 21:27:19.588672 30048 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Committing config change with OpId 5.34: config changed from index 31 to 34, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:19.593475 28903 catalog_manager.cc:5484] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index 31 to 34, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:19.593268 29552 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 31 to 34, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:19.592748 29001 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 31 to 34, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:19.601212 29476 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 31 to 34, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:19.603847 29157 consensus_queue.cc:237] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 32, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:19.605430 29552 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240502 21:27:19.605408 29476 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240502 21:27:19.605898 30048 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 21:27:19.606086 30075 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 21:27:19.608633 30048 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Committing config change with OpId 5.35: config changed from index 34 to 35, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:19.609158 29476 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 5 FOLLOWER]: Committing config change with OpId 5.35: config changed from index 34 to 35, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:19.609798 29552 raft_consensus.cc:2932] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Committing config change with OpId 5.35: config changed from index 34 to 35, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:19.613092 28890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 31c63a7053fa45b69eba6d7f7e78139b with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:19.613710 28895 catalog_manager.cc:5484] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index 34 to 35, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New cstate: current_term: 5 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:19.617331 28994 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c63a7053fa45b69eba6d7f7e78139b with delete_type TABLET_DATA_TOMBSTONED (TS 94f26f2c271544729391496f0fa81a58 not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:52236
I20240502 21:27:19.617950 30080 tablet_replica.cc:330] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: stopping tablet replica
I20240502 21:27:19.618132 30080 raft_consensus.cc:2229] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:19.618453 30080 raft_consensus.cc:2258] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:19.619210 30080 ts_tablet_manager.cc:1875] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:19.623829 30080 ts_tablet_manager.cc:1888] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.34
I20240502 21:27:19.623919 30080 log.cc:1196] T 31c63a7053fa45b69eba6d7f7e78139b P 94f26f2c271544729391496f0fa81a58: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-0-root/wals/31c63a7053fa45b69eba6d7f7e78139b
I20240502 21:27:19.624504 28889 catalog_manager.cc:4830] TS 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): tablet 31c63a7053fa45b69eba6d7f7e78139b (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:19.962476 29476 raft_consensus.cc:1203] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.31->[4.32-4.32] Dedup: 4.32->[]
I20240502 21:27:19.975114 30082 raft_consensus.cc:1050] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: attempting to promote NON_VOTER 9c60776a34a54a09822291687d162a23 to VOTER
I20240502 21:27:19.975893 30082 consensus_queue.cc:237] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:19.977945 29552 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23 [term 2 LEARNER]: Refusing update from remote peer c0c2fc14113d41c498fd6abcbbbd03c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 21:27:19.978240 29157 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer c0c2fc14113d41c498fd6abcbbbd03c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 21:27:19.978466 29081 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer c0c2fc14113d41c498fd6abcbbbd03c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 21:27:19.978960 29987 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:19.979226 30081 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:19.979936 30081 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:19.982806 30084 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:19.983317 29081 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:19.984041 29157 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:19.986356 29552 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:19.988395 28904 catalog_manager.cc:5484] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 reported cstate change: config changed from index 30 to 31, 9c60776a34a54a09822291687d162a23 (127.25.16.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:19.993062 29402 consensus_queue.cc:237] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:19.994550 29081 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer c0c2fc14113d41c498fd6abcbbbd03c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 21:27:19.994550 29552 raft_consensus.cc:1261] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Refusing update from remote peer c0c2fc14113d41c498fd6abcbbbd03c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240502 21:27:19.995010 30084 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:19.995222 29987 consensus_queue.cc:1035] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:19.997202 30081 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:19.997777 29552 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:20.000392 28947 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ccd9ffc92c5c47d088c530e919f2ff77 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:20.000684 28903 catalog_manager.cc:5484] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 reported cstate change: config changed from index 31 to 32, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New cstate: current_term: 2 leader_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:20.000701 29081 raft_consensus.cc:2932] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:20.003998 29138 tablet_service.cc:1514] Processing DeleteTablet for tablet ccd9ffc92c5c47d088c530e919f2ff77 with delete_type TABLET_DATA_TOMBSTONED (TS 5f1d764da7eb4f0c9fd4cabdca685d7b not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:34466
I20240502 21:27:20.004882 30088 tablet_replica.cc:330] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b: stopping tablet replica
I20240502 21:27:20.005075 30088 raft_consensus.cc:2229] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:20.005259 30088 raft_consensus.cc:2258] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:20.006207 30088 ts_tablet_manager.cc:1875] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:20.010560 30088 ts_tablet_manager.cc:1888] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240502 21:27:20.010656 30088 log.cc:1196] T ccd9ffc92c5c47d088c530e919f2ff77 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-2-root/wals/ccd9ffc92c5c47d088c530e919f2ff77
I20240502 21:27:20.011217 28890 catalog_manager.cc:4830] TS 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): tablet ccd9ffc92c5c47d088c530e919f2ff77 (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:20.290195 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:20.293138 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
new_leader_uuid: "9c60776a34a54a09822291687d162a23"
from {username='slave'} at 127.0.0.1:42850
I20240502 21:27:20.293335 29157 raft_consensus.cc:595] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Received request to transfer leadership to TS 9c60776a34a54a09822291687d162a23
I20240502 21:27:20.294274 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7ce57dd3a92645839765ba1ce6609430"
dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
mode: GRACEFUL
new_leader_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:42850
I20240502 21:27:20.294453 29157 raft_consensus.cc:595] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Received request to transfer leadership to TS 94f26f2c271544729391496f0fa81a58
I20240502 21:27:20.294798 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 21:27:20.295010 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:20.336167 30067 raft_consensus.cc:979] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b: : Instructing follower 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:20.336333 30067 raft_consensus.cc:1067] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Signalling peer 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:20.336890 29001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ce57dd3a92645839765ba1ce6609430"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:52250
I20240502 21:27:20.337069 29001 raft_consensus.cc:482] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:20.337164 29001 raft_consensus.cc:3037] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:20.339694 29001 raft_consensus.cc:504] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:20.340309 29001 leader_election.cc:288] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 2 election: Requested vote from peers 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785), 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791)
I20240502 21:27:20.340656 29552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce57dd3a92645839765ba1ce6609430" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "9c60776a34a54a09822291687d162a23"
I20240502 21:27:20.340682 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce57dd3a92645839765ba1ce6609430" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:20.340781 29552 raft_consensus.cc:3037] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:20.340823 29157 raft_consensus.cc:3032] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:20.340871 29157 raft_consensus.cc:729] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:20.341055 29157 consensus_queue.cc:260] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:20.341264 29157 raft_consensus.cc:3037] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:20.342501 29552 raft_consensus.cc:2454] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 2.
I20240502 21:27:20.343051 29157 raft_consensus.cc:2454] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 2.
I20240502 21:27:20.343391 28972 leader_election.cc:302] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 94f26f2c271544729391496f0fa81a58; no voters:
I20240502 21:27:20.343696 30061 raft_consensus.cc:2781] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:20.344019 30061 raft_consensus.cc:686] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:20.344220 30061 consensus_queue.cc:237] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:20.346330 28903 catalog_manager.cc:5484] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 1 to 2, leader changed from 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 2 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:20.441092 30091 raft_consensus.cc:1050] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58: attempting to promote NON_VOTER 501feae33ef0484f956ca7a1e090ae01 to VOTER
I20240502 21:27:20.441886 30091 consensus_queue.cc:237] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 28, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:20.444072 29476 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 4 LEARNER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 21:27:20.444196 29081 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 21:27:20.444196 29402 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240502 21:27:20.444725 30089 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 21:27:20.445003 30093 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 21:27:20.445331 30091 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 21:27:20.449855 30089 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:20.450362 29476 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:20.454005 28903 catalog_manager.cc:5484] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 reported cstate change: config changed from index 32 to 33, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:20.455168 29402 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:20.456789 29081 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:20.459471 29003 consensus_queue.cc:237] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:20.460937 29476 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240502 21:27:20.461504 30089 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 21:27:20.462018 29402 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240502 21:27:20.462929 30097 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 21:27:20.463687 30093 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:20.464196 29476 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:20.468050 28889 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d08f3643db34c8ba4d4391b630be366 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:20.469067 28903 catalog_manager.cc:5484] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index 33 to 34, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New cstate: current_term: 4 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:20.469336 29401 raft_consensus.cc:2932] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 9846ea932cb14dff98f20d67f1829791 (127.25.16.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } }
I20240502 21:27:20.472224 29071 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d08f3643db34c8ba4d4391b630be366 with delete_type TABLET_DATA_TOMBSTONED (TS 9846ea932cb14dff98f20d67f1829791 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:51584
I20240502 21:27:20.472913 30098 tablet_replica.cc:330] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791: stopping tablet replica
I20240502 21:27:20.473099 30098 raft_consensus.cc:2229] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:20.473428 30098 raft_consensus.cc:2258] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:20.474294 30098 ts_tablet_manager.cc:1875] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:20.478190 30098 ts_tablet_manager.cc:1888] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.33
I20240502 21:27:20.478272 30098 log.cc:1196] T 6d08f3643db34c8ba4d4391b630be366 P 9846ea932cb14dff98f20d67f1829791: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-1-root/wals/6d08f3643db34c8ba4d4391b630be366
I20240502 21:27:20.478811 28890 catalog_manager.cc:4830] TS 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): tablet 6d08f3643db34c8ba4d4391b630be366 (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:20.734810 30075 raft_consensus.cc:979] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: : Instructing follower 9c60776a34a54a09822291687d162a23 to start an election
I20240502 21:27:20.734966 30075 raft_consensus.cc:1067] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Signalling peer 9c60776a34a54a09822291687d162a23 to start an election
I20240502 21:27:20.735711 29552 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b"
dest_uuid: "9c60776a34a54a09822291687d162a23"
from {username='slave'} at 127.0.0.1:39912
I20240502 21:27:20.735890 29552 raft_consensus.cc:482] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:20.735962 29552 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:20.737792 29552 raft_consensus.cc:504] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:20.738426 29552 leader_election.cc:288] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 6 election: Requested vote from peers 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785), 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297)
I20240502 21:27:20.738819 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "9c60776a34a54a09822291687d162a23" candidate_term: 6 candidate_status { last_received { term: 5 index: 35 } } ignore_live_leader: true dest_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b"
I20240502 21:27:20.739020 29157 raft_consensus.cc:3032] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:20.739090 29157 raft_consensus.cc:729] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5f1d764da7eb4f0c9fd4cabdca685d7b, State: Running, Role: LEADER
I20240502 21:27:20.739349 29157 consensus_queue.cc:260] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.35, Last appended by leader: 35, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:20.739729 29157 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:20.742583 29157 raft_consensus.cc:2454] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c60776a34a54a09822291687d162a23 in term 6.
I20240502 21:27:20.748693 29517 leader_election.cc:302] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9c60776a34a54a09822291687d162a23; no voters:
I20240502 21:27:20.748976 29932 raft_consensus.cc:2781] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:20.749341 29932 raft_consensus.cc:686] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 6 LEADER]: Becoming Leader. State: Replica: 9c60776a34a54a09822291687d162a23, State: Running, Role: LEADER
I20240502 21:27:20.749580 29932 consensus_queue.cc:237] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.35, Last appended by leader: 35, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:20.752048 28903 catalog_manager.cc:5484] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 reported cstate change: term changed from 5 to 6, leader changed from 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) to 9c60776a34a54a09822291687d162a23 (127.25.16.70). New cstate: current_term: 6 leader_uuid: "9c60776a34a54a09822291687d162a23" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:20.753752 29476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c63a7053fa45b69eba6d7f7e78139b" candidate_uuid: "9c60776a34a54a09822291687d162a23" candidate_term: 6 candidate_status { last_received { term: 5 index: 35 } } ignore_live_leader: true dest_uuid: "501feae33ef0484f956ca7a1e090ae01"
I20240502 21:27:20.754024 29476 raft_consensus.cc:3037] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:20.755746 29476 raft_consensus.cc:2454] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c60776a34a54a09822291687d162a23 in term 6.
I20240502 21:27:20.853289 29552 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 21:27:20.853749 30097 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:20.857348 29157 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 21:27:20.857933 30089 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:21.226636 29157 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 6 FOLLOWER]: Refusing update from remote peer 9c60776a34a54a09822291687d162a23: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 6 index: 36. (index mismatch)
I20240502 21:27:21.227131 29932 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 21:27:21.231156 29476 raft_consensus.cc:1261] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 6 FOLLOWER]: Refusing update from remote peer 9c60776a34a54a09822291687d162a23: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 6 index: 36. (index mismatch)
I20240502 21:27:21.232087 29932 consensus_queue.cc:1035] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 21:27:21.474442 29001 consensus_queue.cc:237] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
W20240502 21:27:21.476342 28970 consensus_peers.cc:485] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 -> Peer c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499): Couldn't send request to peer c0c2fc14113d41c498fd6abcbbbd03c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ce57dd3a92645839765ba1ce6609430. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:21.476518 29552 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 21:27:21.476914 29157 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240502 21:27:21.476936 30113 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:21.477366 30112 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 21:27:21.478757 30113 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:21.479264 29552 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:21.482342 29157 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:21.482537 28895 catalog_manager.cc:5484] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index 27 to 29, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New cstate: current_term: 2 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:21.484565 29150 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 29, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:21.490626 29552 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240502 21:27:21.491617 29081 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240502 21:27:21.492121 29001 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240502 21:27:21.492450 30118 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:21.492856 30075 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:21.493125 30118 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 21:27:21.495592 30119 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Committing config change with OpId 5.32: config changed from index 31 to 32. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:21.496124 29001 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 31 to 32. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:21.496189 29552 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 31 to 32. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:21.499332 28904 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index 31 to 32. New cstate: current_term: 5 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:21.499692 29081 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 31 to 32. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } attrs { replace: true } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:21.503142 29157 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:21.504784 29552 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240502 21:27:21.504837 29081 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240502 21:27:21.505369 30119 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 21:27:21.505609 30075 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 21:27:21.507828 30075 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:21.509054 29081 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:21.509054 29552 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:21.510699 28890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea5a9bdae3f3459f915eaf3bdd3d346c with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:21.512370 28904 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index 32 to 33, VOTER 94f26f2c271544729391496f0fa81a58 (127.25.16.65) evicted. New cstate: current_term: 5 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:21.516686 28994 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5a9bdae3f3459f915eaf3bdd3d346c with delete_type TABLET_DATA_TOMBSTONED (TS 94f26f2c271544729391496f0fa81a58 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:52236
I20240502 21:27:21.518585 29157 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:21.521723 29081 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20240502 21:27:21.522336 30118 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 21:27:21.522363 29552 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20240502 21:27:21.524631 30075 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Committing config change with OpId 5.34: config changed from index 33 to 34, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:21.525206 29081 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:21.525699 30119 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 21:27:21.529420 28890 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ea5a9bdae3f3459f915eaf3bdd3d346c with cas_config_opid_index 33: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:27:21.529680 28895 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index 33 to 34, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New cstate: current_term: 5 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:21.530217 29552 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34, NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } } }
I20240502 21:27:21.537211 30123 tablet_replica.cc:330] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58: stopping tablet replica
I20240502 21:27:21.537420 30123 raft_consensus.cc:2229] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:21.537671 30123 raft_consensus.cc:2258] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:21.538368 30123 ts_tablet_manager.cc:1875] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:27:21.542510 29120 consensus_peers.cc:485] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b -> Peer c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499): Couldn't send request to peer c0c2fc14113d41c498fd6abcbbbd03c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea5a9bdae3f3459f915eaf3bdd3d346c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:21.543385 30123 ts_tablet_manager.cc:1888] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.32
I20240502 21:27:21.543462 30123 log.cc:1196] T ea5a9bdae3f3459f915eaf3bdd3d346c P 94f26f2c271544729391496f0fa81a58: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-0-root/wals/ea5a9bdae3f3459f915eaf3bdd3d346c
I20240502 21:27:21.543943 28889 catalog_manager.cc:4830] TS 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): tablet ea5a9bdae3f3459f915eaf3bdd3d346c (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:21.874095 30127 ts_tablet_manager.cc:902] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7: Initiating tablet copy from peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159)
I20240502 21:27:21.875372 30127 tablet_copy_client.cc:308] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.65:33159
I20240502 21:27:21.875960 29017 tablet_copy_service.cc:140] P 94f26f2c271544729391496f0fa81a58: Received BeginTabletCopySession request for tablet 7ce57dd3a92645839765ba1ce6609430 from peer c0c2fc14113d41c498fd6abcbbbd03c7 ({username='slave'} at 127.0.0.1:54710)
I20240502 21:27:21.876120 29017 tablet_copy_service.cc:161] P 94f26f2c271544729391496f0fa81a58: Beginning new tablet copy session on tablet 7ce57dd3a92645839765ba1ce6609430 from peer c0c2fc14113d41c498fd6abcbbbd03c7 at {username='slave'} at 127.0.0.1:54710: session id = c0c2fc14113d41c498fd6abcbbbd03c7-7ce57dd3a92645839765ba1ce6609430
I20240502 21:27:21.878307 29017 tablet_copy_source_session.cc:205] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:21.879299 30127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce57dd3a92645839765ba1ce6609430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.884114 30127 tablet_copy_client.cc:783] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:21.884353 30127 tablet_copy_client.cc:647] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:21.885637 30131 tablet_copy_client.cc:1002] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:22.078642 30135 ts_tablet_manager.cc:902] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: Initiating tablet copy from peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:22.079766 30135 tablet_copy_client.cc:308] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.67:41785
I20240502 21:27:22.080348 29167 tablet_copy_service.cc:140] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Received BeginTabletCopySession request for tablet ea5a9bdae3f3459f915eaf3bdd3d346c from peer c0c2fc14113d41c498fd6abcbbbd03c7 ({username='slave'} at 127.0.0.1:42876)
I20240502 21:27:22.080466 29167 tablet_copy_service.cc:161] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Beginning new tablet copy session on tablet ea5a9bdae3f3459f915eaf3bdd3d346c from peer c0c2fc14113d41c498fd6abcbbbd03c7 at {username='slave'} at 127.0.0.1:42876: session id = c0c2fc14113d41c498fd6abcbbbd03c7-ea5a9bdae3f3459f915eaf3bdd3d346c
I20240502 21:27:22.083017 29167 tablet_copy_source_session.cc:205] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:22.083972 30135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5a9bdae3f3459f915eaf3bdd3d346c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.088788 30135 tablet_copy_client.cc:783] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:22.089018 30135 tablet_copy_client.cc:647] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:22.090289 30139 tablet_copy_client.cc:1002] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:22.295322 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:22.298677 29003 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54a27d02080d420382add870508aaad4"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
new_leader_uuid: "9c60776a34a54a09822291687d162a23"
from {username='slave'} at 127.0.0.1:52268
I20240502 21:27:22.298902 29003 raft_consensus.cc:595] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Received request to transfer leadership to TS 9c60776a34a54a09822291687d162a23
I20240502 21:27:22.299337 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 21:27:22.299554 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:22.537057 30140 raft_consensus.cc:979] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58: : Instructing follower 9c60776a34a54a09822291687d162a23 to start an election
I20240502 21:27:22.537228 30140 raft_consensus.cc:1067] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Signalling peer 9c60776a34a54a09822291687d162a23 to start an election
I20240502 21:27:22.537808 29552 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54a27d02080d420382add870508aaad4"
dest_uuid: "9c60776a34a54a09822291687d162a23"
from {username='slave'} at 127.0.0.1:56374
I20240502 21:27:22.537998 29552 raft_consensus.cc:482] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:22.538091 29552 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:22.540199 29552 raft_consensus.cc:504] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:22.541288 29003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a27d02080d420382add870508aaad4" candidate_uuid: "9c60776a34a54a09822291687d162a23" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:22.541347 29401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a27d02080d420382add870508aaad4" candidate_uuid: "9c60776a34a54a09822291687d162a23" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7"
I20240502 21:27:22.541491 29003 raft_consensus.cc:3032] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:22.541491 29401 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:22.541579 29003 raft_consensus.cc:729] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:22.541811 29003 consensus_queue.cc:260] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:22.542111 29003 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:22.544088 29401 raft_consensus.cc:2454] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c60776a34a54a09822291687d162a23 in term 4.
I20240502 21:27:22.544435 29515 leader_election.cc:302] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c60776a34a54a09822291687d162a23, c0c2fc14113d41c498fd6abcbbbd03c7; no voters:
I20240502 21:27:22.544672 29932 raft_consensus.cc:2781] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:22.544922 29003 raft_consensus.cc:2454] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c60776a34a54a09822291687d162a23 in term 4.
I20240502 21:27:22.545033 29932 raft_consensus.cc:686] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 4 LEADER]: Becoming Leader. State: Replica: 9c60776a34a54a09822291687d162a23, State: Running, Role: LEADER
I20240502 21:27:22.541162 29552 leader_election.cc:288] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 4 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499)
I20240502 21:27:22.545313 29932 consensus_queue.cc:237] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:22.547467 28904 catalog_manager.cc:5484] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 reported cstate change: term changed from 3 to 4, leader changed from 94f26f2c271544729391496f0fa81a58 (127.25.16.65) to 9c60776a34a54a09822291687d162a23 (127.25.16.70). New cstate: current_term: 4 leader_uuid: "9c60776a34a54a09822291687d162a23" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:23.035790 29401 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Refusing update from remote peer 9c60776a34a54a09822291687d162a23: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240502 21:27:23.036310 29932 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 21:27:23.039759 29003 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Refusing update from remote peer 9c60776a34a54a09822291687d162a23: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240502 21:27:23.061259 29932 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240502 21:27:23.885948 30127 tablet_copy_client.cc:523] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:23.889478 30127 tablet_bootstrap.cc:492] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap starting.
I20240502 21:27:23.919929 30127 tablet_bootstrap.cc:492] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:23.920347 30127 tablet_bootstrap.cc:492] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap complete.
I20240502 21:27:23.920591 30127 ts_tablet_manager.cc:1367] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent bootstrapping tablet: real 0.031s user 0.030s sys 0.002s
I20240502 21:27:23.921213 30127 raft_consensus.cc:348] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:23.921449 30127 raft_consensus.cc:729] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c0c2fc14113d41c498fd6abcbbbd03c7, State: Initialized, Role: LEARNER
I20240502 21:27:23.921674 30127 consensus_queue.cc:260] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:23.922191 30127 ts_tablet_manager.cc:1398] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:23.922847 29017 tablet_copy_service.cc:342] P 94f26f2c271544729391496f0fa81a58: Request end of tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-7ce57dd3a92645839765ba1ce6609430 received from {username='slave'} at 127.0.0.1:54710
I20240502 21:27:23.922981 29017 tablet_copy_service.cc:433] P 94f26f2c271544729391496f0fa81a58: ending tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-7ce57dd3a92645839765ba1ce6609430 on tablet 7ce57dd3a92645839765ba1ce6609430 with peer c0c2fc14113d41c498fd6abcbbbd03c7
I20240502 21:27:24.090626 30135 tablet_copy_client.cc:523] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:24.094044 30135 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap starting.
I20240502 21:27:24.124248 29401 raft_consensus.cc:1203] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20240502 21:27:24.133071 30135 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:24.133424 30135 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: Bootstrap complete.
I20240502 21:27:24.133677 30135 ts_tablet_manager.cc:1367] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent bootstrapping tablet: real 0.040s user 0.033s sys 0.004s
I20240502 21:27:24.134397 30135 raft_consensus.cc:348] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:24.134599 30135 raft_consensus.cc:729] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: c0c2fc14113d41c498fd6abcbbbd03c7, State: Initialized, Role: LEARNER
I20240502 21:27:24.134788 30135 consensus_queue.cc:260] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: true } }
I20240502 21:27:24.135370 30135 ts_tablet_manager.cc:1398] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:24.136062 29167 tablet_copy_service.cc:342] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Request end of tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-ea5a9bdae3f3459f915eaf3bdd3d346c received from {username='slave'} at 127.0.0.1:42876
I20240502 21:27:24.136171 29167 tablet_copy_service.cc:433] P 5f1d764da7eb4f0c9fd4cabdca685d7b: ending tablet copy session c0c2fc14113d41c498fd6abcbbbd03c7-ea5a9bdae3f3459f915eaf3bdd3d346c on tablet ea5a9bdae3f3459f915eaf3bdd3d346c with peer c0c2fc14113d41c498fd6abcbbbd03c7
I20240502 21:27:24.299897 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:24.308565 29552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54a27d02080d420382add870508aaad4"
dest_uuid: "9c60776a34a54a09822291687d162a23"
mode: GRACEFUL
new_leader_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:52924
I20240502 21:27:24.308779 29552 raft_consensus.cc:595] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 4 LEADER]: Received request to transfer leadership to TS 94f26f2c271544729391496f0fa81a58
I20240502 21:27:24.309182 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 21:27:24.309391 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:24.594692 30163 raft_consensus.cc:1050] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58: attempting to promote NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 to VOTER
I20240502 21:27:24.595577 30163 consensus_queue.cc:237] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:24.597532 29552 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 21:27:24.597841 29157 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 21:27:24.597975 29401 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEARNER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240502 21:27:24.598047 30163 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:24.598577 30150 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:24.598861 30162 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240502 21:27:24.602041 30150 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:24.602645 29157 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:24.605357 29552 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:24.607738 28904 catalog_manager.cc:5484] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index 29 to 30, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:24.614701 29401 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:24.624255 29003 consensus_queue.cc:237] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:24.625659 29401 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 21:27:24.625735 29552 raft_consensus.cc:1261] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240502 21:27:24.626107 30163 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:24.626334 30162 consensus_queue.cc:1035] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 21:27:24.628044 30163 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:24.628835 29552 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:24.631888 29401 raft_consensus.cc:2932] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:24.632504 28903 catalog_manager.cc:5484] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 reported cstate change: config changed from index 30 to 31, VOTER 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67) evicted. New cstate: current_term: 2 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:24.634567 28889 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7ce57dd3a92645839765ba1ce6609430 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:24.635988 29138 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ce57dd3a92645839765ba1ce6609430 with delete_type TABLET_DATA_TOMBSTONED (TS 5f1d764da7eb4f0c9fd4cabdca685d7b not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:34466
I20240502 21:27:24.636595 30172 tablet_replica.cc:330] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b: stopping tablet replica
I20240502 21:27:24.636796 30172 raft_consensus.cc:2229] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:24.637034 30172 raft_consensus.cc:2258] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:24.638041 30172 ts_tablet_manager.cc:1875] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:24.639823 30173 raft_consensus.cc:979] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: : Instructing follower 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:24.639953 30173 raft_consensus.cc:1067] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 4 LEADER]: Signalling peer 94f26f2c271544729391496f0fa81a58 to start an election
I20240502 21:27:24.640547 29001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54a27d02080d420382add870508aaad4"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
from {username='slave'} at 127.0.0.1:48054
I20240502 21:27:24.640726 29001 raft_consensus.cc:482] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:24.640817 29001 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:24.643280 29001 raft_consensus.cc:504] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:24.643611 30172 ts_tablet_manager.cc:1888] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240502 21:27:24.643719 30172 log.cc:1196] T 7ce57dd3a92645839765ba1ce6609430 P 5f1d764da7eb4f0c9fd4cabdca685d7b: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-2-root/wals/7ce57dd3a92645839765ba1ce6609430
I20240502 21:27:24.643988 29001 leader_election.cc:288] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 5 election: Requested vote from peers 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791), c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499)
I20240502 21:27:24.644212 29552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a27d02080d420382add870508aaad4" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "9c60776a34a54a09822291687d162a23"
I20240502 21:27:24.644362 29401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a27d02080d420382add870508aaad4" candidate_uuid: "94f26f2c271544729391496f0fa81a58" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7"
I20240502 21:27:24.644420 29552 raft_consensus.cc:3032] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:27:24.644503 29552 raft_consensus.cc:729] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c60776a34a54a09822291687d162a23, State: Running, Role: LEADER
I20240502 21:27:24.644529 29401 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:24.644665 29552 consensus_queue.cc:260] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:24.644929 29552 raft_consensus.cc:3037] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:24.645730 29402 raft_consensus.cc:1203] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.33->[5.34-5.34] Dedup: 5.34->[]
I20240502 21:27:24.646606 29401 raft_consensus.cc:2454] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 5.
I20240502 21:27:24.647125 28970 leader_election.cc:302] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f26f2c271544729391496f0fa81a58, c0c2fc14113d41c498fd6abcbbbd03c7; no voters:
I20240502 21:27:24.647363 30150 raft_consensus.cc:2781] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:24.647475 30150 raft_consensus.cc:686] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 5 LEADER]: Becoming Leader. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:24.647523 29552 raft_consensus.cc:2454] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f26f2c271544729391496f0fa81a58 in term 5.
I20240502 21:27:24.647792 30150 consensus_queue.cc:237] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:24.648373 28890 catalog_manager.cc:4830] TS 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): tablet 7ce57dd3a92645839765ba1ce6609430 (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:24.649920 28895 catalog_manager.cc:5484] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 reported cstate change: term changed from 4 to 5, leader changed from 9c60776a34a54a09822291687d162a23 (127.25.16.70) to 94f26f2c271544729391496f0fa81a58 (127.25.16.65). New cstate: current_term: 5 leader_uuid: "94f26f2c271544729391496f0fa81a58" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:25.087489 29552 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240502 21:27:25.087991 30150 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 21:27:25.091400 29401 raft_consensus.cc:1261] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Refusing update from remote peer 94f26f2c271544729391496f0fa81a58: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240502 21:27:25.091934 30150 consensus_queue.cc:1035] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 21:27:25.175096 30180 raft_consensus.cc:1050] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b: attempting to promote NON_VOTER c0c2fc14113d41c498fd6abcbbbd03c7 to VOTER
I20240502 21:27:25.175992 30180 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 29, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:25.177886 29401 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 LEARNER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240502 21:27:25.178027 29081 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240502 21:27:25.178122 29552 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240502 21:27:25.178443 30149 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 21:27:25.178710 30181 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 21:27:25.178917 30180 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 21:27:25.185005 30149 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Committing config change with OpId 5.35: config changed from index 34 to 35, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:25.185618 29401 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Committing config change with OpId 5.35: config changed from index 34 to 35, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:25.185886 29552 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Committing config change with OpId 5.35: config changed from index 34 to 35, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:25.189313 29081 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Committing config change with OpId 5.35: config changed from index 34 to 35, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } }
I20240502 21:27:25.190318 28895 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index 34 to 35, c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:25.639947 29157 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.35, Last appended by leader: 29, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false replace: true } }
I20240502 21:27:25.641688 29081 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20240502 21:27:25.641728 29552 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20240502 21:27:25.641728 29401 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20240502 21:27:25.642170 30180 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 21:27:25.642450 30149 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 21:27:25.642668 30183 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 21:27:25.645220 30183 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Committing config change with OpId 5.36: config changed from index 35 to 36. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false replace: true } } }
I20240502 21:27:25.645790 29081 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Committing config change with OpId 5.36: config changed from index 35 to 36. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false replace: true } } }
I20240502 21:27:25.648743 29552 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Committing config change with OpId 5.36: config changed from index 35 to 36. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false replace: true } } }
I20240502 21:27:25.650260 28903 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index 35 to 36. New cstate: current_term: 5 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } }
I20240502 21:27:25.650419 29401 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Committing config change with OpId 5.36: config changed from index 35 to 36. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false replace: true } } }
I20240502 21:27:25.655264 29157 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 5.36, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } }
I20240502 21:27:25.658108 29552 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20240502 21:27:25.658335 29081 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20240502 21:27:25.658607 30180 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240502 21:27:25.659394 30180 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240502 21:27:25.660607 30181 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Committing config change with OpId 5.37: config changed from index 36 to 37, VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:25.661196 29552 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 36 to 37, VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:25.663884 28890 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea5a9bdae3f3459f915eaf3bdd3d346c with cas_config_opid_index 36: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:25.664592 28904 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b reported cstate change: config changed from index 36 to 37, VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) evicted. New cstate: current_term: 5 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:25.665807 29081 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 36 to 37, VOTER c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } }
I20240502 21:27:25.669632 29157 consensus_queue.cc:237] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 5.37, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:25.672214 29081 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 37. Preceding OpId from leader: term: 5 index: 38. (index mismatch)
I20240502 21:27:25.672246 29552 raft_consensus.cc:1261] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Refusing update from remote peer 5f1d764da7eb4f0c9fd4cabdca685d7b: Log matching property violated. Preceding OpId in replica: term: 5 index: 37. Preceding OpId from leader: term: 5 index: 38. (index mismatch)
I20240502 21:27:25.672696 30183 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
W20240502 21:27:25.673134 29122 consensus_peers.cc:485] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b -> Peer 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297): Couldn't send request to peer 501feae33ef0484f956ca7a1e090ae01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea5a9bdae3f3459f915eaf3bdd3d346c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:25.673709 30183 consensus_queue.cc:1035] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240502 21:27:25.675022 30149 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Committing config change with OpId 5.38: config changed from index 37 to 38, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:25.675781 29392 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5a9bdae3f3459f915eaf3bdd3d346c with delete_type TABLET_DATA_TOMBSTONED (TS c0c2fc14113d41c498fd6abcbbbd03c7 not found in new config with opid_index 37) from {username='slave'} at 127.0.0.1:52052
I20240502 21:27:25.675597 29076 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Committing config change with OpId 5.38: config changed from index 37 to 38, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:25.676376 30193 tablet_replica.cc:330] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: stopping tablet replica
I20240502 21:27:25.676544 30193 raft_consensus.cc:2229] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:25.676766 30193 raft_consensus.cc:2258] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:25.677673 30193 ts_tablet_manager.cc:1875] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:25.678746 28890 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ea5a9bdae3f3459f915eaf3bdd3d346c with cas_config_opid_index 37: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:27:25.679450 28904 catalog_manager.cc:5484] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index 37 to 38, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New cstate: current_term: 5 leader_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:25.680132 29552 raft_consensus.cc:2932] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Committing config change with OpId 5.38: config changed from index 37 to 38, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:25.684639 30193 ts_tablet_manager.cc:1888] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.36
I20240502 21:27:25.684763 30193 log.cc:1196] T ea5a9bdae3f3459f915eaf3bdd3d346c P c0c2fc14113d41c498fd6abcbbbd03c7: Deleting WAL directory at /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714685126025081-25665-0/minicluster-data/ts-3-root/wals/ea5a9bdae3f3459f915eaf3bdd3d346c
I20240502 21:27:25.685413 28888 catalog_manager.cc:4830] TS c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499): tablet ea5a9bdae3f3459f915eaf3bdd3d346c (table test-workload [id=8e5cfa6205934648b1194dc4f1a7f6d9]) successfully deleted
I20240502 21:27:26.156409 30194 ts_tablet_manager.cc:902] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01: Initiating tablet copy from peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
I20240502 21:27:26.157449 30194 tablet_copy_client.cc:308] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.67:41785
I20240502 21:27:26.158041 29167 tablet_copy_service.cc:140] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Received BeginTabletCopySession request for tablet ea5a9bdae3f3459f915eaf3bdd3d346c from peer 501feae33ef0484f956ca7a1e090ae01 ({username='slave'} at 127.0.0.1:42866)
I20240502 21:27:26.158188 29167 tablet_copy_service.cc:161] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Beginning new tablet copy session on tablet ea5a9bdae3f3459f915eaf3bdd3d346c from peer 501feae33ef0484f956ca7a1e090ae01 at {username='slave'} at 127.0.0.1:42866: session id = 501feae33ef0484f956ca7a1e090ae01-ea5a9bdae3f3459f915eaf3bdd3d346c
I20240502 21:27:26.160108 29167 tablet_copy_source_session.cc:205] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:26.161157 30194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5a9bdae3f3459f915eaf3bdd3d346c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.165660 30194 tablet_copy_client.cc:783] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:26.166087 30194 tablet_copy_client.cc:647] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:26.167435 30198 tablet_copy_client.cc:1002] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:26.309734 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:26.312381 29001 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366"
dest_uuid: "94f26f2c271544729391496f0fa81a58"
mode: GRACEFUL
new_leader_uuid: "501feae33ef0484f956ca7a1e090ae01"
from {username='slave'} at 127.0.0.1:52268
I20240502 21:27:26.312582 29001 raft_consensus.cc:595] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Received request to transfer leadership to TS 501feae33ef0484f956ca7a1e090ae01
I20240502 21:27:26.313019 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 21:27:26.313252 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 21:27:26.671177 28953 auto_rebalancer.cc:254] failed to send replica move request: Invalid argument: Server with UUID 501feae33ef0484f956ca7a1e090ae01 is already a member of the config. RaftConfig: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
W20240502 21:27:26.672499 28953 auto_rebalancer.cc:663] Could not move replica: Illegal state: ea5a9bdae3f3459f915eaf3bdd3d346c: source replica 9846ea932cb14dff98f20d67f1829791 does not have REPLACE attribute set
W20240502 21:27:26.672575 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: ea5a9bdae3f3459f915eaf3bdd3d346c: source replica 9846ea932cb14dff98f20d67f1829791 does not have REPLACE attribute set
I20240502 21:27:26.811904 30201 raft_consensus.cc:979] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58: : Instructing follower 501feae33ef0484f956ca7a1e090ae01 to start an election
I20240502 21:27:26.812065 30201 raft_consensus.cc:1067] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Signalling peer 501feae33ef0484f956ca7a1e090ae01 to start an election
I20240502 21:27:26.812649 29476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366"
dest_uuid: "501feae33ef0484f956ca7a1e090ae01"
from {username='slave'} at 127.0.0.1:33708
I20240502 21:27:26.812808 29476 raft_consensus.cc:482] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:26.812876 29476 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:26.814641 29476 raft_consensus.cc:504] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:26.815235 29476 leader_election.cc:288] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [CANDIDATE]: Term 5 election: Requested vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499)
I20240502 21:27:26.815680 29401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "501feae33ef0484f956ca7a1e090ae01" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7"
I20240502 21:27:26.815857 29401 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:26.815798 29001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08f3643db34c8ba4d4391b630be366" candidate_uuid: "501feae33ef0484f956ca7a1e090ae01" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "94f26f2c271544729391496f0fa81a58"
I20240502 21:27:26.816030 29001 raft_consensus.cc:3032] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:27:26.816090 29001 raft_consensus.cc:729] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 94f26f2c271544729391496f0fa81a58, State: Running, Role: LEADER
I20240502 21:27:26.816362 29001 consensus_queue.cc:260] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:26.816660 29001 raft_consensus.cc:3037] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:26.817793 29401 raft_consensus.cc:2454] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501feae33ef0484f956ca7a1e090ae01 in term 5.
I20240502 21:27:26.818143 29440 leader_election.cc:302] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 501feae33ef0484f956ca7a1e090ae01, c0c2fc14113d41c498fd6abcbbbd03c7; no voters:
I20240502 21:27:26.818348 30190 raft_consensus.cc:2781] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:26.818536 29001 raft_consensus.cc:2454] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501feae33ef0484f956ca7a1e090ae01 in term 5.
I20240502 21:27:26.818697 30190 raft_consensus.cc:686] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 5 LEADER]: Becoming Leader. State: Replica: 501feae33ef0484f956ca7a1e090ae01, State: Running, Role: LEADER
I20240502 21:27:26.818930 30190 consensus_queue.cc:237] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:26.821164 28904 catalog_manager.cc:5484] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 reported cstate change: term changed from 4 to 5, leader changed from 94f26f2c271544729391496f0fa81a58 (127.25.16.65) to 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69). New cstate: current_term: 5 leader_uuid: "501feae33ef0484f956ca7a1e090ae01" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:27.370445 29401 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Refusing update from remote peer 501feae33ef0484f956ca7a1e090ae01: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240502 21:27:27.371074 30190 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 21:27:27.375032 29001 raft_consensus.cc:1261] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Refusing update from remote peer 501feae33ef0484f956ca7a1e090ae01: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240502 21:27:27.375598 30190 consensus_queue.cc:1035] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 21:27:27.679252 29552 consensus_queue.cc:237] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:27.681780 29076 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Refusing update from remote peer 9c60776a34a54a09822291687d162a23: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
W20240502 21:27:27.682039 29517 consensus_peers.cc:485] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 -> Peer 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297): Couldn't send request to peer 501feae33ef0484f956ca7a1e090ae01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1de7fda467344968ab81f653b8501210. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:27.682049 29157 raft_consensus.cc:1261] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Refusing update from remote peer 9c60776a34a54a09822291687d162a23: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240502 21:27:27.682317 30173 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:27.682585 30216 consensus_queue.cc:1035] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 21:27:27.686139 30216 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 26 to 28, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:27.689796 29076 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 26 to 28, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:27.693794 28902 catalog_manager.cc:5484] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 reported cstate change: config changed from index 26 to 28, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New cstate: current_term: 2 leader_uuid: "9c60776a34a54a09822291687d162a23" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:27.697154 29150 raft_consensus.cc:2932] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 26 to 28, NON_VOTER 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } } }
I20240502 21:27:28.168442 30194 tablet_copy_client.cc:523] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:28.171787 30194 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01: Bootstrap starting.
I20240502 21:27:28.208987 30194 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:28.209375 30194 tablet_bootstrap.cc:492] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01: Bootstrap complete.
I20240502 21:27:28.209611 30194 ts_tablet_manager.cc:1367] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01: Time spent bootstrapping tablet: real 0.038s user 0.037s sys 0.000s
I20240502 21:27:28.210328 30194 raft_consensus.cc:348] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:28.210573 30194 raft_consensus.cc:729] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 501feae33ef0484f956ca7a1e090ae01, State: Initialized, Role: LEARNER
I20240502 21:27:28.210765 30194 consensus_queue.cc:260] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 5.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:28.211236 30194 ts_tablet_manager.cc:1398] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:28.211968 29167 tablet_copy_service.cc:342] P 5f1d764da7eb4f0c9fd4cabdca685d7b: Request end of tablet copy session 501feae33ef0484f956ca7a1e090ae01-ea5a9bdae3f3459f915eaf3bdd3d346c received from {username='slave'} at 127.0.0.1:42866
I20240502 21:27:28.212059 29167 tablet_copy_service.cc:433] P 5f1d764da7eb4f0c9fd4cabdca685d7b: ending tablet copy session 501feae33ef0484f956ca7a1e090ae01-ea5a9bdae3f3459f915eaf3bdd3d346c on tablet ea5a9bdae3f3459f915eaf3bdd3d346c with peer 501feae33ef0484f956ca7a1e090ae01
I20240502 21:27:28.223157 30194 ts_tablet_manager.cc:902] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01: Initiating tablet copy from peer 9c60776a34a54a09822291687d162a23 (127.25.16.70:35791)
I20240502 21:27:28.224220 30194 tablet_copy_client.cc:308] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Beginning tablet copy session from remote peer at address 127.25.16.70:35791
I20240502 21:27:28.245371 29562 tablet_copy_service.cc:140] P 9c60776a34a54a09822291687d162a23: Received BeginTabletCopySession request for tablet 1de7fda467344968ab81f653b8501210 from peer 501feae33ef0484f956ca7a1e090ae01 ({username='slave'} at 127.0.0.1:52928)
I20240502 21:27:28.245486 29562 tablet_copy_service.cc:161] P 9c60776a34a54a09822291687d162a23: Beginning new tablet copy session on tablet 1de7fda467344968ab81f653b8501210 from peer 501feae33ef0484f956ca7a1e090ae01 at {username='slave'} at 127.0.0.1:52928: session id = 501feae33ef0484f956ca7a1e090ae01-1de7fda467344968ab81f653b8501210
I20240502 21:27:28.247431 29562 tablet_copy_source_session.cc:205] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:28.248374 30194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de7fda467344968ab81f653b8501210. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.252521 30194 tablet_copy_client.cc:783] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:28.252746 30194 tablet_copy_client.cc:647] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:28.253947 30228 tablet_copy_client.cc:1002] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Injecting latency into file download: 2000
I20240502 21:27:28.313583 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:28.315920 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:27:28.316133 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:28.589802 29476 raft_consensus.cc:1203] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.37->[5.38-5.38] Dedup: 5.38->[]
I20240502 21:27:28.971716 25665 tablet_server.cc:178] TabletServer@127.25.16.65:0 shutting down...
I20240502 21:27:28.981942 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:27:28.982213 25665 tablet_replica.cc:330] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58: stopping tablet replica
I20240502 21:27:28.982460 25665 raft_consensus.cc:2229] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 5 LEADER]: Raft consensus shutting down.
I20240502 21:27:28.982842 25665 raft_consensus.cc:2258] T 54a27d02080d420382add870508aaad4 P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:28.983518 25665 tablet_replica.cc:330] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58: stopping tablet replica
I20240502 21:27:28.983703 25665 raft_consensus.cc:2229] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:28.983965 25665 raft_consensus.cc:2258] T 93477dec2c574c49a0bc9a7f022ad7fa P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:28.984599 25665 tablet_replica.cc:330] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58: stopping tablet replica
I20240502 21:27:28.984781 25665 raft_consensus.cc:2229] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:28.984927 25665 raft_consensus.cc:2258] T 6d08f3643db34c8ba4d4391b630be366 P 94f26f2c271544729391496f0fa81a58 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:28.985461 25665 tablet_replica.cc:330] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58: stopping tablet replica
I20240502 21:27:28.985581 25665 raft_consensus.cc:2229] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:28.985873 25665 raft_consensus.cc:2258] T af41f02157c94f55b2bc5090f566592e P 94f26f2c271544729391496f0fa81a58 [term 4 FOLLOWER]: Raft consensus is shut down!
W20240502 21:27:28.986266 29441 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 21:27:28.986342 25665 tablet_replica.cc:330] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58: stopping tablet replica
I20240502 21:27:28.986474 25665 raft_consensus.cc:2229] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 2 LEADER]: Raft consensus shutting down.
I20240502 21:27:28.986816 25665 raft_consensus.cc:2258] T 7ce57dd3a92645839765ba1ce6609430 P 94f26f2c271544729391496f0fa81a58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:28.987388 25665 tablet_replica.cc:330] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58: stopping tablet replica
W20240502 21:27:28.987507 29441 consensus_peers.cc:485] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 -> Peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): Couldn't send request to peer 94f26f2c271544729391496f0fa81a58. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.987560 25665 raft_consensus.cc:2229] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:28.987710 25665 raft_consensus.cc:2258] T ee4eba5acb8d4af986a00f937de0270e P 94f26f2c271544729391496f0fa81a58 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240502 21:27:29.001446 29366 consensus_peers.cc:485] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 -> Peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): Couldn't send request to peer 94f26f2c271544729391496f0fa81a58. Status: Network error: Client connection negotiation failed: client connection to 127.25.16.65:33159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:29.005658 25665 tablet_server.cc:194] TabletServer@127.25.16.65:0 shutdown complete.
I20240502 21:27:29.012501 25665 tablet_server.cc:178] TabletServer@127.25.16.66:0 shutting down...
I20240502 21:27:29.037657 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:27:29.037902 25665 tablet_replica.cc:330] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791: stopping tablet replica
I20240502 21:27:29.038101 25665 raft_consensus.cc:2229] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 4 LEADER]: Raft consensus shutting down.
I20240502 21:27:29.038439 25665 raft_consensus.cc:2258] T ab35657135e44445aee4aa6961f70c27 P 9846ea932cb14dff98f20d67f1829791 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.039168 25665 tablet_replica.cc:330] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791: stopping tablet replica
I20240502 21:27:29.039324 25665 raft_consensus.cc:2229] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.039532 25665 raft_consensus.cc:2258] T 93477dec2c574c49a0bc9a7f022ad7fa P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.040166 25665 tablet_replica.cc:330] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791: stopping tablet replica
I20240502 21:27:29.040378 25665 raft_consensus.cc:2229] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.040688 25665 raft_consensus.cc:2258] T ccd9ffc92c5c47d088c530e919f2ff77 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 21:27:29.040906 29121 consensus_peers.cc:485] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b -> Peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): Couldn't send request to peer 94f26f2c271544729391496f0fa81a58. Status: Network error: Client connection negotiation failed: client connection to 127.25.16.65:33159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:29.041301 25665 tablet_replica.cc:330] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791: stopping tablet replica
I20240502 21:27:29.041448 25665 raft_consensus.cc:2229] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 3 LEADER]: Raft consensus shutting down.
I20240502 21:27:29.041707 25665 raft_consensus.cc:2258] T 5010a09a5243452dad7cd9b90535a229 P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.042196 25665 tablet_replica.cc:330] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791: stopping tablet replica
I20240502 21:27:29.042344 25665 raft_consensus.cc:2229] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.042533 25665 raft_consensus.cc:2258] T 1de7fda467344968ab81f653b8501210 P 9846ea932cb14dff98f20d67f1829791 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.042989 25665 tablet_replica.cc:330] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791: stopping tablet replica
I20240502 21:27:29.043125 25665 raft_consensus.cc:2229] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.043332 25665 raft_consensus.cc:2258] T ee4eba5acb8d4af986a00f937de0270e P 9846ea932cb14dff98f20d67f1829791 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.043813 25665 tablet_replica.cc:330] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791: stopping tablet replica
I20240502 21:27:29.043933 25665 raft_consensus.cc:2229] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.044178 25665 raft_consensus.cc:2258] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9846ea932cb14dff98f20d67f1829791 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.053304 25665 tablet_server.cc:194] TabletServer@127.25.16.66:0 shutdown complete.
W20240502 21:27:29.060936 29517 consensus_peers.cc:485] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 -> Peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): Couldn't send request to peer 9846ea932cb14dff98f20d67f1829791. Status: Network error: Client connection negotiation failed: client connection to 127.25.16.66:46109: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:29.061501 25665 tablet_server.cc:178] TabletServer@127.25.16.67:0 shutting down...
I20240502 21:27:29.072263 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:27:29.072564 25665 tablet_replica.cc:330] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b: stopping tablet replica
I20240502 21:27:29.072789 25665 raft_consensus.cc:2229] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 LEADER]: Raft consensus shutting down.
I20240502 21:27:29.073200 25665 raft_consensus.cc:2258] T ea5a9bdae3f3459f915eaf3bdd3d346c P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.073863 25665 tablet_replica.cc:330] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b: stopping tablet replica
I20240502 21:27:29.074005 25665 raft_consensus.cc:2229] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.074290 25665 raft_consensus.cc:2258] T 1de7fda467344968ab81f653b8501210 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.074807 25665 tablet_replica.cc:330] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b: stopping tablet replica
I20240502 21:27:29.074959 25665 raft_consensus.cc:2229] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.075174 25665 raft_consensus.cc:2258] T 5010a09a5243452dad7cd9b90535a229 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.075690 25665 tablet_replica.cc:330] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b: stopping tablet replica
I20240502 21:27:29.075817 25665 raft_consensus.cc:2229] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 LEADER]: Raft consensus shutting down.
I20240502 21:27:29.076102 25665 raft_consensus.cc:2258] T 93477dec2c574c49a0bc9a7f022ad7fa P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.076622 25665 tablet_replica.cc:330] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b: stopping tablet replica
I20240502 21:27:29.076731 25665 raft_consensus.cc:2229] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 6 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.076864 25665 raft_consensus.cc:2258] T 31c63a7053fa45b69eba6d7f7e78139b P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 6 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.077387 25665 tablet_replica.cc:330] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b: stopping tablet replica
I20240502 21:27:29.077509 25665 raft_consensus.cc:2229] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.077642 25665 raft_consensus.cc:2258] T ab35657135e44445aee4aa6961f70c27 P 5f1d764da7eb4f0c9fd4cabdca685d7b [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.102434 25665 tablet_server.cc:194] TabletServer@127.25.16.67:0 shutdown complete.
I20240502 21:27:29.107900 25665 tablet_server.cc:178] TabletServer@127.25.16.68:0 shutting down...
I20240502 21:27:29.117923 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:27:29.118129 25665 tablet_replica.cc:330] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7: stopping tablet replica
I20240502 21:27:29.118342 25665 raft_consensus.cc:2229] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 LEADER]: Raft consensus shutting down.
I20240502 21:27:29.118646 25665 raft_consensus.cc:2258] T ccd9ffc92c5c47d088c530e919f2ff77 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.119206 25665 tablet_replica.cc:330] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7: stopping tablet replica
I20240502 21:27:29.119351 25665 raft_consensus.cc:2229] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 LEADER]: Raft consensus shutting down.
I20240502 21:27:29.119639 25665 raft_consensus.cc:2258] T ee4eba5acb8d4af986a00f937de0270e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.120131 25665 tablet_replica.cc:330] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7: stopping tablet replica
I20240502 21:27:29.120267 25665 raft_consensus.cc:2229] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.120430 25665 raft_consensus.cc:2258] T 54a27d02080d420382add870508aaad4 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.120914 25665 tablet_replica.cc:330] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7: stopping tablet replica
I20240502 21:27:29.121038 25665 raft_consensus.cc:2229] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.121374 25665 raft_consensus.cc:2258] T 6d08f3643db34c8ba4d4391b630be366 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.121944 25665 tablet_replica.cc:330] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7: stopping tablet replica
I20240502 21:27:29.122159 25665 raft_consensus.cc:2229] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.122392 25665 raft_consensus.cc:2258] T af41f02157c94f55b2bc5090f566592e P c0c2fc14113d41c498fd6abcbbbd03c7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.122964 25665 tablet_replica.cc:330] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7: stopping tablet replica
I20240502 21:27:29.123087 25665 raft_consensus.cc:2229] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.123313 25665 raft_consensus.cc:2258] T 7ce57dd3a92645839765ba1ce6609430 P c0c2fc14113d41c498fd6abcbbbd03c7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.138852 25665 tablet_server.cc:194] TabletServer@127.25.16.68:0 shutdown complete.
I20240502 21:27:29.144376 25665 tablet_server.cc:178] TabletServer@127.25.16.69:0 shutting down...
I20240502 21:27:29.154254 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240502 21:27:29.193522 29440 consensus_peers.cc:485] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 -> Peer c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499): Couldn't send request to peer c0c2fc14113d41c498fd6abcbbbd03c7. Status: Network error: Client connection negotiation failed: client connection to 127.25.16.68:45499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:29.279639 29517 consensus_peers.cc:485] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 -> Peer 501feae33ef0484f956ca7a1e090ae01 (127.25.16.69:35297): Couldn't send request to peer 501feae33ef0484f956ca7a1e090ae01. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:29.296881 29441 consensus_peers.cc:485] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 -> Peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): Couldn't send request to peer 94f26f2c271544729391496f0fa81a58. Status: Network error: Client connection negotiation failed: client connection to 127.25.16.65:33159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:29.340816 29517 consensus_peers.cc:485] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 -> Peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): Couldn't send request to peer 5f1d764da7eb4f0c9fd4cabdca685d7b. Status: Network error: Client connection negotiation failed: client connection to 127.25.16.67:41785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:29.342026 29517 consensus_peers.cc:485] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 -> Peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): Couldn't send request to peer 5f1d764da7eb4f0c9fd4cabdca685d7b. Status: Network error: Client connection negotiation failed: client connection to 127.25.16.67:41785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:29.472950 29440 consensus_peers.cc:485] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 -> Peer c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499): Couldn't send request to peer c0c2fc14113d41c498fd6abcbbbd03c7. Status: Network error: Client connection negotiation failed: client connection to 127.25.16.68:45499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:30.069545 30244 raft_consensus.cc:482] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94f26f2c271544729391496f0fa81a58)
I20240502 21:27:30.069746 30244 raft_consensus.cc:504] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:30.070396 30244 leader_election.cc:288] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499)
W20240502 21:27:30.071956 29516 leader_election.cc:334] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): Network error: Client connection negotiation failed: client connection to 127.25.16.65:33159: connect: Connection refused (error 111)
W20240502 21:27:30.072552 29515 leader_election.cc:334] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499): Network error: Client connection negotiation failed: client connection to 127.25.16.68:45499: connect: Connection refused (error 111)
I20240502 21:27:30.072647 29515 leader_election.cc:302] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c60776a34a54a09822291687d162a23; no voters: 94f26f2c271544729391496f0fa81a58, c0c2fc14113d41c498fd6abcbbbd03c7
I20240502 21:27:30.072901 30244 raft_consensus.cc:2726] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:27:30.073240 30241 raft_consensus.cc:482] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9846ea932cb14dff98f20d67f1829791)
I20240502 21:27:30.073359 30241 raft_consensus.cc:504] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:30.074090 30241 leader_election.cc:288] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785), 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109)
W20240502 21:27:30.077263 29442 leader_election.cc:334] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): Network error: Client connection negotiation failed: client connection to 127.25.16.67:41785: connect: Connection refused (error 111)
W20240502 21:27:30.077430 29442 leader_election.cc:334] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): Network error: Client connection negotiation failed: client connection to 127.25.16.66:46109: connect: Connection refused (error 111)
I20240502 21:27:30.077569 29442 leader_election.cc:302] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 501feae33ef0484f956ca7a1e090ae01; no voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791
I20240502 21:27:30.077785 30241 raft_consensus.cc:2726] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 21:27:30.146656 30241 raft_consensus.cc:482] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9846ea932cb14dff98f20d67f1829791)
I20240502 21:27:30.146808 30241 raft_consensus.cc:504] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: false } }
I20240502 21:27:30.147416 30241 leader_election.cc:288] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
W20240502 21:27:30.148715 29442 leader_election.cc:334] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): Network error: Client connection negotiation failed: client connection to 127.25.16.66:46109: connect: Connection refused (error 111)
W20240502 21:27:30.148890 29442 leader_election.cc:334] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): Network error: Client connection negotiation failed: client connection to 127.25.16.67:41785: connect: Connection refused (error 111)
I20240502 21:27:30.148975 29442 leader_election.cc:302] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 501feae33ef0484f956ca7a1e090ae01; no voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791
I20240502 21:27:30.149188 30241 raft_consensus.cc:2726] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240502 21:27:30.172704 30244 raft_consensus.cc:482] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f1d764da7eb4f0c9fd4cabdca685d7b)
I20240502 21:27:30.172828 30244 raft_consensus.cc:504] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } attrs { replace: true } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:30.173372 30244 leader_election.cc:288] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109), 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785)
W20240502 21:27:30.174644 29517 leader_election.cc:334] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9846ea932cb14dff98f20d67f1829791 (127.25.16.66:46109): Network error: Client connection negotiation failed: client connection to 127.25.16.66:46109: connect: Connection refused (error 111)
W20240502 21:27:30.174822 29517 leader_election.cc:334] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5f1d764da7eb4f0c9fd4cabdca685d7b (127.25.16.67:41785): Network error: Client connection negotiation failed: client connection to 127.25.16.67:41785: connect: Connection refused (error 111)
I20240502 21:27:30.174892 29517 leader_election.cc:302] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c60776a34a54a09822291687d162a23; no voters: 5f1d764da7eb4f0c9fd4cabdca685d7b, 9846ea932cb14dff98f20d67f1829791
I20240502 21:27:30.175076 30244 raft_consensus.cc:2726] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240502 21:27:30.254290 30194 tablet_copy_client.cc:523] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:30.257664 30194 tablet_bootstrap.cc:492] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01: Bootstrap starting.
I20240502 21:27:30.291901 30194 tablet_bootstrap.cc:492] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=91 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:30.292238 30194 tablet_bootstrap.cc:492] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01: Bootstrap complete.
I20240502 21:27:30.292502 30194 ts_tablet_manager.cc:1367] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.011s
I20240502 21:27:30.293084 30194 raft_consensus.cc:348] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:30.293310 30194 raft_consensus.cc:729] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 501feae33ef0484f956ca7a1e090ae01, State: Initialized, Role: LEARNER
I20240502 21:27:30.293499 30194 consensus_queue.cc:260] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9846ea932cb14dff98f20d67f1829791" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 46109 } attrs { replace: true } } peers { permanent_uuid: "5f1d764da7eb4f0c9fd4cabdca685d7b" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 41785 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "501feae33ef0484f956ca7a1e090ae01" member_type: NON_VOTER last_known_addr { host: "127.25.16.69" port: 35297 } attrs { promote: true } }
I20240502 21:27:30.294030 30194 ts_tablet_manager.cc:1398] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:30.294533 29562 tablet_copy_service.cc:342] P 9c60776a34a54a09822291687d162a23: Request end of tablet copy session 501feae33ef0484f956ca7a1e090ae01-1de7fda467344968ab81f653b8501210 received from {username='slave'} at 127.0.0.1:52928
I20240502 21:27:30.294628 29562 tablet_copy_service.cc:433] P 9c60776a34a54a09822291687d162a23: ending tablet copy session 501feae33ef0484f956ca7a1e090ae01-1de7fda467344968ab81f653b8501210 on tablet 1de7fda467344968ab81f653b8501210 with peer 501feae33ef0484f956ca7a1e090ae01
I20240502 21:27:30.295675 25665 tablet_replica.cc:330] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01: stopping tablet replica
I20240502 21:27:30.295902 25665 raft_consensus.cc:2229] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Raft consensus shutting down.
I20240502 21:27:30.296068 25665 raft_consensus.cc:2258] T 1de7fda467344968ab81f653b8501210 P 501feae33ef0484f956ca7a1e090ae01 [term 2 LEARNER]: Raft consensus is shut down!
I20240502 21:27:30.296936 25665 tablet_replica.cc:330] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01: stopping tablet replica
I20240502 21:27:30.297230 25665 raft_consensus.cc:2229] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.297650 25665 raft_consensus.cc:2258] T ab35657135e44445aee4aa6961f70c27 P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.298528 25665 tablet_replica.cc:330] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01: stopping tablet replica
I20240502 21:27:30.298712 25665 raft_consensus.cc:2229] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 4 LEADER]: Raft consensus shutting down.
I20240502 21:27:30.299089 25665 raft_consensus.cc:2258] T af41f02157c94f55b2bc5090f566592e P 501feae33ef0484f956ca7a1e090ae01 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.299538 25665 tablet_replica.cc:330] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01: stopping tablet replica
I20240502 21:27:30.299683 25665 raft_consensus.cc:2229] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01 [term 5 LEARNER]: Raft consensus shutting down.
I20240502 21:27:30.299897 25665 raft_consensus.cc:2258] T ea5a9bdae3f3459f915eaf3bdd3d346c P 501feae33ef0484f956ca7a1e090ae01 [term 5 LEARNER]: Raft consensus is shut down!
I20240502 21:27:30.300549 25665 tablet_replica.cc:330] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01: stopping tablet replica
I20240502 21:27:30.300742 25665 raft_consensus.cc:2229] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 6 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.301016 25665 raft_consensus.cc:2258] T 31c63a7053fa45b69eba6d7f7e78139b P 501feae33ef0484f956ca7a1e090ae01 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.301779 25665 tablet_replica.cc:330] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01: stopping tablet replica
I20240502 21:27:30.301980 25665 raft_consensus.cc:2229] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.302235 25665 raft_consensus.cc:2258] T 5010a09a5243452dad7cd9b90535a229 P 501feae33ef0484f956ca7a1e090ae01 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.303025 25665 tablet_replica.cc:330] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01: stopping tablet replica
I20240502 21:27:30.303205 25665 raft_consensus.cc:2229] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 5 LEADER]: Raft consensus shutting down.
I20240502 21:27:30.303593 25665 raft_consensus.cc:2258] T 6d08f3643db34c8ba4d4391b630be366 P 501feae33ef0484f956ca7a1e090ae01 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.315167 30244 raft_consensus.cc:482] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 94f26f2c271544729391496f0fa81a58)
I20240502 21:27:30.315352 30244 raft_consensus.cc:504] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94f26f2c271544729391496f0fa81a58" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 33159 } } peers { permanent_uuid: "9c60776a34a54a09822291687d162a23" member_type: VOTER last_known_addr { host: "127.25.16.70" port: 35791 } attrs { promote: false } } peers { permanent_uuid: "c0c2fc14113d41c498fd6abcbbbd03c7" member_type: VOTER last_known_addr { host: "127.25.16.68" port: 45499 } attrs { promote: false } }
I20240502 21:27:30.316222 30244 leader_election.cc:288] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159), c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499)
I20240502 21:27:30.316886 28959 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:30.320492 28959 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:27:30.320695 28959 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 21:27:30.321053 29515 leader_election.cc:334] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c0c2fc14113d41c498fd6abcbbbd03c7 (127.25.16.68:45499): Network error: Client connection negotiation failed: client connection to 127.25.16.68:45499: connect: Connection refused (error 111)
W20240502 21:27:30.321200 29516 leader_election.cc:334] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 94f26f2c271544729391496f0fa81a58 (127.25.16.65:33159): Network error: Client connection negotiation failed: client connection to 127.25.16.65:33159: connect: Connection refused (error 111)
I20240502 21:27:30.321301 29516 leader_election.cc:302] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c60776a34a54a09822291687d162a23; no voters: 94f26f2c271544729391496f0fa81a58, c0c2fc14113d41c498fd6abcbbbd03c7
I20240502 21:27:30.321511 30244 raft_consensus.cc:2726] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240502 21:27:30.343030 25665 tablet_server.cc:194] TabletServer@127.25.16.69:0 shutdown complete.
I20240502 21:27:30.347738 25665 tablet_server.cc:178] TabletServer@127.25.16.70:0 shutting down...
W20240502 21:27:30.350732 28953 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 21:27:30.350817 28953 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 21:27:30.356906 25665 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:27:30.357090 25665 tablet_replica.cc:330] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23: stopping tablet replica
I20240502 21:27:30.357259 25665 raft_consensus.cc:2229] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.357470 25665 raft_consensus.cc:2258] T ccd9ffc92c5c47d088c530e919f2ff77 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.357977 25665 tablet_replica.cc:330] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23: stopping tablet replica
I20240502 21:27:30.358111 25665 raft_consensus.cc:2229] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 6 LEADER]: Raft consensus shutting down.
I20240502 21:27:30.358352 25665 raft_consensus.cc:2258] T 31c63a7053fa45b69eba6d7f7e78139b P 9c60776a34a54a09822291687d162a23 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.358772 25665 tablet_replica.cc:330] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23: stopping tablet replica
I20240502 21:27:30.358898 25665 raft_consensus.cc:2229] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.359243 25665 raft_consensus.cc:2258] T ea5a9bdae3f3459f915eaf3bdd3d346c P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.359656 25665 tablet_replica.cc:330] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23: stopping tablet replica
I20240502 21:27:30.359773 25665 raft_consensus.cc:2229] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.359946 25665 raft_consensus.cc:2258] T 7ce57dd3a92645839765ba1ce6609430 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.360397 25665 tablet_replica.cc:330] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23: stopping tablet replica
I20240502 21:27:30.360512 25665 raft_consensus.cc:2229] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 2 LEADER]: Raft consensus shutting down.
I20240502 21:27:30.360803 25665 raft_consensus.cc:2258] T 1de7fda467344968ab81f653b8501210 P 9c60776a34a54a09822291687d162a23 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.361205 25665 tablet_replica.cc:330] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23: stopping tablet replica
I20240502 21:27:30.361316 25665 raft_consensus.cc:2229] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.361438 25665 raft_consensus.cc:2258] T 54a27d02080d420382add870508aaad4 P 9c60776a34a54a09822291687d162a23 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.365759 25665 tablet_server.cc:194] TabletServer@127.25.16.70:0 shutdown complete.
I20240502 21:27:30.369563 25665 master.cc:537] Master@127.25.16.126:42491 shutting down...
I20240502 21:27:30.377291 25665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:30.377450 25665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.377522 25665 tablet_replica.cc:330] T 00000000000000000000000000000000 P 481b5308c76d43678924338449b5a9f6: stopping tablet replica
I20240502 21:27:30.390138 25665 master.cc:552] Master@127.25.16.126:42491 shutdown complete.
[ OK ] AutoRebalancerTest.TestMaxMovesPerServer (32402 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240502 21:27:30.428202 25665 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.16.126:39399
I20240502 21:27:30.428691 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:30.431171 30249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:30.431610 30250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:30.432543 25665 server_base.cc:1034] running on GCE node
W20240502 21:27:30.433197 30252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:30.433470 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:27:30.433523 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:27:30.433554 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685250433547 us; error 0 us; skew 500 ppm
I20240502 21:27:30.433670 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:30.445782 25665 webserver.cc:460] Webserver started at http://127.25.16.126:37699/ using document root <none> and password file <none>
I20240502 21:27:30.445983 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:30.446031 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:30.446122 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:30.446722 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/master-0-root/instance:
uuid: "f90f2b1ef55c4fbf9cacae867c1feed1"
format_stamp: "Formatted at 2024-05-02 21:27:30 on dist-test-slave-b524"
I20240502 21:27:30.449306 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:30.450747 30257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:30.451048 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:30.451139 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/master-0-root
uuid: "f90f2b1ef55c4fbf9cacae867c1feed1"
format_stamp: "Formatted at 2024-05-02 21:27:30 on dist-test-slave-b524"
I20240502 21:27:30.451238 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:30.468416 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:30.468925 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:30.478559 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.126:39399
I20240502 21:27:30.478578 30308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.126:39399 every 8 connection(s)
I20240502 21:27:30.479893 30309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.483723 30309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1: Bootstrap starting.
I20240502 21:27:30.485011 30309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:30.486389 30309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1: No bootstrap required, opened a new log
I20240502 21:27:30.486886 30309 raft_consensus.cc:348] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90f2b1ef55c4fbf9cacae867c1feed1" member_type: VOTER }
I20240502 21:27:30.487010 30309 raft_consensus.cc:374] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:30.487047 30309 raft_consensus.cc:729] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90f2b1ef55c4fbf9cacae867c1feed1, State: Initialized, Role: FOLLOWER
I20240502 21:27:30.487211 30309 consensus_queue.cc:260] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90f2b1ef55c4fbf9cacae867c1feed1" member_type: VOTER }
I20240502 21:27:30.487323 30309 raft_consensus.cc:388] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:27:30.487362 30309 raft_consensus.cc:482] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:27:30.487411 30309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:30.489024 30309 raft_consensus.cc:504] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90f2b1ef55c4fbf9cacae867c1feed1" member_type: VOTER }
I20240502 21:27:30.489185 30309 leader_election.cc:302] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f90f2b1ef55c4fbf9cacae867c1feed1; no voters:
I20240502 21:27:30.489493 30309 leader_election.cc:288] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:27:30.489602 30312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:30.489985 30312 raft_consensus.cc:686] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [term 1 LEADER]: Becoming Leader. State: Replica: f90f2b1ef55c4fbf9cacae867c1feed1, State: Running, Role: LEADER
I20240502 21:27:30.490204 30309 sys_catalog.cc:564] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:27:30.490170 30312 consensus_queue.cc:237] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90f2b1ef55c4fbf9cacae867c1feed1" member_type: VOTER }
I20240502 21:27:30.490901 30314 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f90f2b1ef55c4fbf9cacae867c1feed1. Latest consensus state: current_term: 1 leader_uuid: "f90f2b1ef55c4fbf9cacae867c1feed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90f2b1ef55c4fbf9cacae867c1feed1" member_type: VOTER } }
I20240502 21:27:30.491051 30314 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [sys.catalog]: This master's current role is: LEADER
I20240502 21:27:30.491060 30313 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f90f2b1ef55c4fbf9cacae867c1feed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90f2b1ef55c4fbf9cacae867c1feed1" member_type: VOTER } }
I20240502 21:27:30.491437 30313 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1 [sys.catalog]: This master's current role is: LEADER
I20240502 21:27:30.492141 30317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:27:30.493381 30317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:27:30.494599 25665 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:27:30.496026 30317 catalog_manager.cc:1347] Generated new cluster ID: 609e2177042844d0b189a73096424ed7
I20240502 21:27:30.496098 30317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:27:30.522114 30317 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:27:30.522830 30317 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:27:30.532207 30317 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f90f2b1ef55c4fbf9cacae867c1feed1: Generated new TSK 0
I20240502 21:27:30.532421 30317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:27:30.559406 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:30.562254 30330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:30.562666 30332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:30.563330 30334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:30.564325 25665 server_base.cc:1034] running on GCE node
I20240502 21:27:30.564538 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:27:30.564589 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:27:30.564635 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685250564629 us; error 0 us; skew 500 ppm
I20240502 21:27:30.564772 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:30.565958 25665 webserver.cc:460] Webserver started at http://127.25.16.65:37815/ using document root <none> and password file <none>
I20240502 21:27:30.566162 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:30.566222 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:30.566341 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:30.566982 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-0-root/instance:
uuid: "039e05b194a046cca2342c6efe73ab87"
format_stamp: "Formatted at 2024-05-02 21:27:30 on dist-test-slave-b524"
I20240502 21:27:30.569422 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:30.570890 30339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:30.571198 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:30.571290 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-0-root
uuid: "039e05b194a046cca2342c6efe73ab87"
format_stamp: "Formatted at 2024-05-02 21:27:30 on dist-test-slave-b524"
I20240502 21:27:30.571386 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:30.596568 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:30.597018 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:30.597489 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:30.598277 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:30.598330 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:30.598388 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:30.598429 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:30.610271 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.65:45931
I20240502 21:27:30.610307 30401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.65:45931 every 8 connection(s)
I20240502 21:27:30.611788 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:30.614970 30407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:30.615350 30408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:30.616144 30410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:30.617692 25665 server_base.cc:1034] running on GCE node
I20240502 21:27:30.617892 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:27:30.617959 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:27:30.617990 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685250617985 us; error 0 us; skew 500 ppm
I20240502 21:27:30.618126 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:30.619293 30402 heartbeater.cc:347] Connected to a master server at 127.25.16.126:39399
I20240502 21:27:30.619411 30402 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:30.619553 25665 webserver.cc:460] Webserver started at http://127.25.16.66:46597/ using document root <none> and password file <none>
I20240502 21:27:30.619731 30402 heartbeater.cc:511] Master 127.25.16.126:39399 requested a full tablet report, sending...
I20240502 21:27:30.619740 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:30.619838 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:30.619946 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:30.620625 30274 ts_manager.cc:194] Registered new tserver with Master: 039e05b194a046cca2342c6efe73ab87 (127.25.16.65:45931)
I20240502 21:27:30.620615 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-1-root/instance:
uuid: "926f90ca0bdc4abcb0a31f4ab996461d"
format_stamp: "Formatted at 2024-05-02 21:27:30 on dist-test-slave-b524"
I20240502 21:27:30.622056 30274 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36280
I20240502 21:27:30.623343 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:30.624747 30415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:30.625072 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:30.625157 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-1-root
uuid: "926f90ca0bdc4abcb0a31f4ab996461d"
format_stamp: "Formatted at 2024-05-02 21:27:30 on dist-test-slave-b524"
I20240502 21:27:30.625244 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:30.634228 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:30.634666 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:30.635120 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:30.635820 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:30.635874 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:30.635932 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:30.635967 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:30.647562 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.66:36101
I20240502 21:27:30.647567 30477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.66:36101 every 8 connection(s)
I20240502 21:27:30.648988 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:30.651811 30481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:30.652222 30482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:30.653055 25665 server_base.cc:1034] running on GCE node
W20240502 21:27:30.653820 30484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:30.654167 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:27:30.654243 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:27:30.654273 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685250654269 us; error 0 us; skew 500 ppm
I20240502 21:27:30.654403 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:30.654906 30478 heartbeater.cc:347] Connected to a master server at 127.25.16.126:39399
I20240502 21:27:30.655038 30478 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:30.655378 30478 heartbeater.cc:511] Master 127.25.16.126:39399 requested a full tablet report, sending...
I20240502 21:27:30.655666 25665 webserver.cc:460] Webserver started at http://127.25.16.67:40623/ using document root <none> and password file <none>
I20240502 21:27:30.655865 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:30.655918 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:30.656014 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:30.656204 30274 ts_manager.cc:194] Registered new tserver with Master: 926f90ca0bdc4abcb0a31f4ab996461d (127.25.16.66:36101)
I20240502 21:27:30.656621 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-2-root/instance:
uuid: "ba0e34a385f748579ad428b1f9233390"
format_stamp: "Formatted at 2024-05-02 21:27:30 on dist-test-slave-b524"
I20240502 21:27:30.657090 30274 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36296
I20240502 21:27:30.659036 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:30.660442 30489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:30.660768 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240502 21:27:30.660859 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-2-root
uuid: "ba0e34a385f748579ad428b1f9233390"
format_stamp: "Formatted at 2024-05-02 21:27:30 on dist-test-slave-b524"
I20240502 21:27:30.660951 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:30.678140 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:30.678543 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:30.679031 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:30.679759 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:30.679812 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:30.679890 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:30.679929 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:30.691649 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.67:33973
I20240502 21:27:30.691668 30552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.67:33973 every 8 connection(s)
I20240502 21:27:30.697698 30553 heartbeater.cc:347] Connected to a master server at 127.25.16.126:39399
I20240502 21:27:30.697798 30553 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:30.698098 30553 heartbeater.cc:511] Master 127.25.16.126:39399 requested a full tablet report, sending...
I20240502 21:27:30.698845 30274 ts_manager.cc:194] Registered new tserver with Master: ba0e34a385f748579ad428b1f9233390 (127.25.16.67:33973)
I20240502 21:27:30.699322 25665 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006801527s
I20240502 21:27:30.699636 30274 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36298
I20240502 21:27:31.623222 30402 heartbeater.cc:503] Master 127.25.16.126:39399 was elected leader, sending a full tablet report...
I20240502 21:27:31.658385 30478 heartbeater.cc:503] Master 127.25.16.126:39399 was elected leader, sending a full tablet report...
I20240502 21:27:31.700763 30553 heartbeater.cc:503] Master 127.25.16.126:39399 was elected leader, sending a full tablet report...
I20240502 21:27:31.726517 25665 test_util.cc:268] Using random seed: 1163177133
I20240502 21:27:31.737515 30274 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36310:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 21:27:31.738147 30274 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:27:31.753863 30516 tablet_service.cc:1467] Processing CreateTablet for tablet a581356fbaf641258d2a92a451eaf30d (DEFAULT_TABLE table=test-workload [id=83da4dbba0c542869e3a97172de71f29]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 21:27:31.753821 30518 tablet_service.cc:1467] Processing CreateTablet for tablet ed648ab2b60944899b549bebb7bc602a (DEFAULT_TABLE table=test-workload [id=83da4dbba0c542869e3a97172de71f29]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 21:27:31.754325 30516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a581356fbaf641258d2a92a451eaf30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.755272 30518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed648ab2b60944899b549bebb7bc602a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.755812 30517 tablet_service.cc:1467] Processing CreateTablet for tablet c3d136a54adf4ec99b4aa1104a714526 (DEFAULT_TABLE table=test-workload [id=83da4dbba0c542869e3a97172de71f29]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 21:27:31.756160 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d136a54adf4ec99b4aa1104a714526. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.757354 30367 tablet_service.cc:1467] Processing CreateTablet for tablet ed648ab2b60944899b549bebb7bc602a (DEFAULT_TABLE table=test-workload [id=83da4dbba0c542869e3a97172de71f29]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 21:27:31.757706 30367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed648ab2b60944899b549bebb7bc602a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.760831 30366 tablet_service.cc:1467] Processing CreateTablet for tablet c3d136a54adf4ec99b4aa1104a714526 (DEFAULT_TABLE table=test-workload [id=83da4dbba0c542869e3a97172de71f29]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 21:27:31.761209 30366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d136a54adf4ec99b4aa1104a714526. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.762706 30365 tablet_service.cc:1467] Processing CreateTablet for tablet a581356fbaf641258d2a92a451eaf30d (DEFAULT_TABLE table=test-workload [id=83da4dbba0c542869e3a97172de71f29]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 21:27:31.763062 30365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a581356fbaf641258d2a92a451eaf30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.764919 30442 tablet_service.cc:1467] Processing CreateTablet for tablet c3d136a54adf4ec99b4aa1104a714526 (DEFAULT_TABLE table=test-workload [id=83da4dbba0c542869e3a97172de71f29]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 21:27:31.765334 30442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d136a54adf4ec99b4aa1104a714526. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.769088 30573 tablet_bootstrap.cc:492] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390: Bootstrap starting.
I20240502 21:27:31.769057 30443 tablet_service.cc:1467] Processing CreateTablet for tablet ed648ab2b60944899b549bebb7bc602a (DEFAULT_TABLE table=test-workload [id=83da4dbba0c542869e3a97172de71f29]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 21:27:31.769472 30441 tablet_service.cc:1467] Processing CreateTablet for tablet a581356fbaf641258d2a92a451eaf30d (DEFAULT_TABLE table=test-workload [id=83da4dbba0c542869e3a97172de71f29]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 21:27:31.769696 30443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed648ab2b60944899b549bebb7bc602a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.769848 30441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a581356fbaf641258d2a92a451eaf30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.777273 30575 tablet_bootstrap.cc:492] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d: Bootstrap starting.
I20240502 21:27:31.778362 30573 tablet_bootstrap.cc:654] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.778597 30575 tablet_bootstrap.cc:654] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.779794 30574 tablet_bootstrap.cc:492] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87: Bootstrap starting.
I20240502 21:27:31.780426 30575 tablet_bootstrap.cc:492] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d: No bootstrap required, opened a new log
I20240502 21:27:31.780557 30575 ts_tablet_manager.cc:1367] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:31.781101 30574 tablet_bootstrap.cc:654] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.781121 30573 tablet_bootstrap.cc:492] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390: No bootstrap required, opened a new log
I20240502 21:27:31.781227 30573 ts_tablet_manager.cc:1367] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 21:27:31.781308 30575 raft_consensus.cc:348] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.781517 30575 raft_consensus.cc:374] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.781632 30575 raft_consensus.cc:729] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926f90ca0bdc4abcb0a31f4ab996461d, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.781844 30575 consensus_queue.cc:260] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.781915 30573 raft_consensus.cc:348] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.782142 30573 raft_consensus.cc:374] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.782245 30573 raft_consensus.cc:729] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0e34a385f748579ad428b1f9233390, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.782471 30573 consensus_queue.cc:260] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.782707 30574 tablet_bootstrap.cc:492] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87: No bootstrap required, opened a new log
I20240502 21:27:31.782821 30574 ts_tablet_manager.cc:1367] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:31.783303 30574 raft_consensus.cc:348] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } }
I20240502 21:27:31.783434 30574 raft_consensus.cc:374] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.783504 30574 raft_consensus.cc:729] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039e05b194a046cca2342c6efe73ab87, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.783720 30574 consensus_queue.cc:260] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } }
I20240502 21:27:31.784384 30574 ts_tablet_manager.cc:1398] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 21:27:31.786159 30574 tablet_bootstrap.cc:492] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87: Bootstrap starting.
I20240502 21:27:31.788208 30574 tablet_bootstrap.cc:654] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.789603 30574 tablet_bootstrap.cc:492] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87: No bootstrap required, opened a new log
I20240502 21:27:31.789716 30574 ts_tablet_manager.cc:1367] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:31.790274 30574 raft_consensus.cc:348] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.790410 30574 raft_consensus.cc:374] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.790484 30574 raft_consensus.cc:729] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039e05b194a046cca2342c6efe73ab87, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.790683 30574 consensus_queue.cc:260] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.791517 30574 ts_tablet_manager.cc:1398] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:31.791741 30574 tablet_bootstrap.cc:492] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87: Bootstrap starting.
I20240502 21:27:31.792878 30575 ts_tablet_manager.cc:1398] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d: Time spent starting tablet: real 0.012s user 0.002s sys 0.010s
I20240502 21:27:31.793294 30573 ts_tablet_manager.cc:1398] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20240502 21:27:31.793537 30573 tablet_bootstrap.cc:492] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390: Bootstrap starting.
I20240502 21:27:31.794785 30575 tablet_bootstrap.cc:492] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d: Bootstrap starting.
I20240502 21:27:31.794910 30573 tablet_bootstrap.cc:654] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.796171 30575 tablet_bootstrap.cc:654] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.797094 30579 raft_consensus.cc:482] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:31.797226 30579 raft_consensus.cc:504] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.798499 30574 tablet_bootstrap.cc:654] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.805047 30573 tablet_bootstrap.cc:492] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390: No bootstrap required, opened a new log
I20240502 21:27:31.805171 30573 ts_tablet_manager.cc:1367] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 21:27:31.805819 30573 raft_consensus.cc:348] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.805999 30573 raft_consensus.cc:374] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.806044 30573 raft_consensus.cc:729] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0e34a385f748579ad428b1f9233390, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.806226 30573 consensus_queue.cc:260] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.806679 30573 ts_tablet_manager.cc:1398] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:31.806912 30573 tablet_bootstrap.cc:492] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390: Bootstrap starting.
I20240502 21:27:31.808142 30574 tablet_bootstrap.cc:492] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87: No bootstrap required, opened a new log
I20240502 21:27:31.808265 30574 ts_tablet_manager.cc:1367] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.006s
I20240502 21:27:31.808321 30573 tablet_bootstrap.cc:654] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.808723 30579 leader_election.cc:288] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039e05b194a046cca2342c6efe73ab87 (127.25.16.65:45931), ba0e34a385f748579ad428b1f9233390 (127.25.16.67:33973)
I20240502 21:27:31.809314 30574 raft_consensus.cc:348] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.809487 30574 raft_consensus.cc:374] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.809537 30574 raft_consensus.cc:729] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039e05b194a046cca2342c6efe73ab87, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.809741 30574 consensus_queue.cc:260] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.812542 30574 ts_tablet_manager.cc:1398] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:27:31.815649 30575 tablet_bootstrap.cc:492] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d: No bootstrap required, opened a new log
I20240502 21:27:31.815776 30575 ts_tablet_manager.cc:1367] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20240502 21:27:31.816469 30575 raft_consensus.cc:348] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } }
I20240502 21:27:31.816664 30575 raft_consensus.cc:374] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.816716 30575 raft_consensus.cc:729] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926f90ca0bdc4abcb0a31f4ab996461d, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.816928 30575 consensus_queue.cc:260] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } }
I20240502 21:27:31.817570 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d136a54adf4ec99b4aa1104a714526" candidate_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039e05b194a046cca2342c6efe73ab87" is_pre_election: true
I20240502 21:27:31.817773 30377 raft_consensus.cc:2454] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 926f90ca0bdc4abcb0a31f4ab996461d in term 0.
I20240502 21:27:31.818259 30418 leader_election.cc:302] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039e05b194a046cca2342c6efe73ab87, 926f90ca0bdc4abcb0a31f4ab996461d; no voters:
I20240502 21:27:31.818549 30583 raft_consensus.cc:2781] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:31.818636 30583 raft_consensus.cc:482] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:31.818725 30583 raft_consensus.cc:3037] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.819386 30575 ts_tablet_manager.cc:1398] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:31.819715 30528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d136a54adf4ec99b4aa1104a714526" candidate_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0e34a385f748579ad428b1f9233390" is_pre_election: true
I20240502 21:27:31.819880 30528 raft_consensus.cc:2454] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 926f90ca0bdc4abcb0a31f4ab996461d in term 0.
I20240502 21:27:31.821046 30583 raft_consensus.cc:504] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.821203 30575 tablet_bootstrap.cc:492] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d: Bootstrap starting.
I20240502 21:27:31.821544 30583 leader_election.cc:288] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [CANDIDATE]: Term 1 election: Requested vote from peers 039e05b194a046cca2342c6efe73ab87 (127.25.16.65:45931), ba0e34a385f748579ad428b1f9233390 (127.25.16.67:33973)
I20240502 21:27:31.821928 30528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d136a54adf4ec99b4aa1104a714526" candidate_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0e34a385f748579ad428b1f9233390"
I20240502 21:27:31.822124 30528 raft_consensus.cc:3037] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.822412 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d136a54adf4ec99b4aa1104a714526" candidate_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039e05b194a046cca2342c6efe73ab87"
I20240502 21:27:31.822551 30377 raft_consensus.cc:3037] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.822860 30575 tablet_bootstrap.cc:654] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.824913 30528 raft_consensus.cc:2454] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 926f90ca0bdc4abcb0a31f4ab996461d in term 1.
I20240502 21:27:31.825062 30377 raft_consensus.cc:2454] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 926f90ca0bdc4abcb0a31f4ab996461d in term 1.
I20240502 21:27:31.825250 30419 leader_election.cc:302] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 926f90ca0bdc4abcb0a31f4ab996461d, ba0e34a385f748579ad428b1f9233390; no voters:
I20240502 21:27:31.825562 30583 raft_consensus.cc:2781] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:31.825680 30583 raft_consensus.cc:686] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 LEADER]: Becoming Leader. State: Replica: 926f90ca0bdc4abcb0a31f4ab996461d, State: Running, Role: LEADER
I20240502 21:27:31.825886 30583 consensus_queue.cc:237] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.826217 30575 tablet_bootstrap.cc:492] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d: No bootstrap required, opened a new log
I20240502 21:27:31.826380 30575 ts_tablet_manager.cc:1367] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 21:27:31.827042 30575 raft_consensus.cc:348] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.827221 30575 raft_consensus.cc:374] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.827266 30575 raft_consensus.cc:729] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926f90ca0bdc4abcb0a31f4ab996461d, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.827788 30575 consensus_queue.cc:260] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.828260 30575 ts_tablet_manager.cc:1398] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:31.829495 30573 tablet_bootstrap.cc:492] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390: No bootstrap required, opened a new log
I20240502 21:27:31.829619 30573 ts_tablet_manager.cc:1367] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.006s
I20240502 21:27:31.829852 30274 catalog_manager.cc:5484] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d reported cstate change: term changed from 0 to 1, leader changed from <none> to 926f90ca0bdc4abcb0a31f4ab996461d (127.25.16.66). New cstate: current_term: 1 leader_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:31.830238 30573 raft_consensus.cc:348] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } }
I20240502 21:27:31.830402 30573 raft_consensus.cc:374] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.830466 30573 raft_consensus.cc:729] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0e34a385f748579ad428b1f9233390, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.830636 30573 consensus_queue.cc:260] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } }
I20240502 21:27:31.831051 30573 ts_tablet_manager.cc:1398] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:31.832953 30581 raft_consensus.cc:482] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:31.833062 30581 raft_consensus.cc:504] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } }
I20240502 21:27:31.833842 30581 leader_election.cc:288] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0e34a385f748579ad428b1f9233390 (127.25.16.67:33973), 926f90ca0bdc4abcb0a31f4ab996461d (127.25.16.66:36101)
I20240502 21:27:31.838974 30528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed648ab2b60944899b549bebb7bc602a" candidate_uuid: "039e05b194a046cca2342c6efe73ab87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0e34a385f748579ad428b1f9233390" is_pre_election: true
I20240502 21:27:31.839104 30528 raft_consensus.cc:2454] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039e05b194a046cca2342c6efe73ab87 in term 0.
I20240502 21:27:31.839159 30453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed648ab2b60944899b549bebb7bc602a" candidate_uuid: "039e05b194a046cca2342c6efe73ab87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" is_pre_election: true
I20240502 21:27:31.839327 30453 raft_consensus.cc:2454] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039e05b194a046cca2342c6efe73ab87 in term 0.
I20240502 21:27:31.839365 30343 leader_election.cc:302] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039e05b194a046cca2342c6efe73ab87, ba0e34a385f748579ad428b1f9233390; no voters:
I20240502 21:27:31.839565 30581 raft_consensus.cc:2781] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:31.839628 30581 raft_consensus.cc:482] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:31.839699 30581 raft_consensus.cc:3037] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.841367 30581 raft_consensus.cc:504] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } }
I20240502 21:27:31.841845 30581 leader_election.cc:288] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [CANDIDATE]: Term 1 election: Requested vote from peers ba0e34a385f748579ad428b1f9233390 (127.25.16.67:33973), 926f90ca0bdc4abcb0a31f4ab996461d (127.25.16.66:36101)
I20240502 21:27:31.842118 30528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed648ab2b60944899b549bebb7bc602a" candidate_uuid: "039e05b194a046cca2342c6efe73ab87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0e34a385f748579ad428b1f9233390"
I20240502 21:27:31.842180 30453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed648ab2b60944899b549bebb7bc602a" candidate_uuid: "039e05b194a046cca2342c6efe73ab87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926f90ca0bdc4abcb0a31f4ab996461d"
I20240502 21:27:31.842272 30528 raft_consensus.cc:3037] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.842285 30453 raft_consensus.cc:3037] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.844870 30528 raft_consensus.cc:2454] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039e05b194a046cca2342c6efe73ab87 in term 1.
I20240502 21:27:31.844870 30453 raft_consensus.cc:2454] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039e05b194a046cca2342c6efe73ab87 in term 1.
I20240502 21:27:31.845242 30343 leader_election.cc:302] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039e05b194a046cca2342c6efe73ab87, ba0e34a385f748579ad428b1f9233390; no voters:
I20240502 21:27:31.845453 30581 raft_consensus.cc:2781] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:31.845775 30581 raft_consensus.cc:686] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [term 1 LEADER]: Becoming Leader. State: Replica: 039e05b194a046cca2342c6efe73ab87, State: Running, Role: LEADER
I20240502 21:27:31.845975 30581 consensus_queue.cc:237] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } }
I20240502 21:27:31.848026 30273 catalog_manager.cc:5484] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039e05b194a046cca2342c6efe73ab87 (127.25.16.65). New cstate: current_term: 1 leader_uuid: "039e05b194a046cca2342c6efe73ab87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:31.853653 30583 raft_consensus.cc:482] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:31.853754 30583 raft_consensus.cc:504] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.854193 30583 leader_election.cc:288] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039e05b194a046cca2342c6efe73ab87 (127.25.16.65:45931), ba0e34a385f748579ad428b1f9233390 (127.25.16.67:33973)
I20240502 21:27:31.854511 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a581356fbaf641258d2a92a451eaf30d" candidate_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039e05b194a046cca2342c6efe73ab87" is_pre_election: true
I20240502 21:27:31.854535 30528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a581356fbaf641258d2a92a451eaf30d" candidate_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0e34a385f748579ad428b1f9233390" is_pre_election: true
I20240502 21:27:31.854665 30377 raft_consensus.cc:2454] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 926f90ca0bdc4abcb0a31f4ab996461d in term 0.
I20240502 21:27:31.854691 30528 raft_consensus.cc:2454] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 926f90ca0bdc4abcb0a31f4ab996461d in term 0.
I20240502 21:27:31.854981 30419 leader_election.cc:302] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 039e05b194a046cca2342c6efe73ab87, 926f90ca0bdc4abcb0a31f4ab996461d, ba0e34a385f748579ad428b1f9233390; no voters:
I20240502 21:27:31.855202 30583 raft_consensus.cc:2781] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:31.855269 30583 raft_consensus.cc:482] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:31.855321 30583 raft_consensus.cc:3037] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.857028 30583 raft_consensus.cc:504] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:31.857497 30583 leader_election.cc:288] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [CANDIDATE]: Term 1 election: Requested vote from peers 039e05b194a046cca2342c6efe73ab87 (127.25.16.65:45931), ba0e34a385f748579ad428b1f9233390 (127.25.16.67:33973)
I20240502 21:27:31.857767 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a581356fbaf641258d2a92a451eaf30d" candidate_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039e05b194a046cca2342c6efe73ab87"
I20240502 21:27:31.857851 30528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a581356fbaf641258d2a92a451eaf30d" candidate_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0e34a385f748579ad428b1f9233390"
I20240502 21:27:31.857926 30377 raft_consensus.cc:3037] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.857985 30528 raft_consensus.cc:3037] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.860478 30528 raft_consensus.cc:2454] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 926f90ca0bdc4abcb0a31f4ab996461d in term 1.
I20240502 21:27:31.860478 30377 raft_consensus.cc:2454] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 926f90ca0bdc4abcb0a31f4ab996461d in term 1.
I20240502 21:27:31.860823 30418 leader_election.cc:302] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039e05b194a046cca2342c6efe73ab87, 926f90ca0bdc4abcb0a31f4ab996461d; no voters:
I20240502 21:27:31.861065 30583 raft_consensus.cc:2781] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:31.861171 30583 raft_consensus.cc:686] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 LEADER]: Becoming Leader. State: Replica: 926f90ca0bdc4abcb0a31f4ab996461d, State: Running, Role: LEADER
I20240502 21:27:31.861359 30583 consensus_queue.cc:237] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
W20240502 21:27:31.862074 30403 tablet.cc:2369] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:27:31.863484 30273 catalog_manager.cc:5484] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d reported cstate change: term changed from 0 to 1, leader changed from <none> to 926f90ca0bdc4abcb0a31f4ab996461d (127.25.16.66). New cstate: current_term: 1 leader_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:31.871412 25665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:31.874178 30598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:31.876195 30599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:31.876271 30601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:31.876470 25665 server_base.cc:1034] running on GCE node
I20240502 21:27:31.876811 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:27:31.876902 25665 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:27:31.876957 25665 hybrid_clock.cc:648] HybridClock initialized: now 1714685251876943 us; error 0 us; skew 500 ppm
I20240502 21:27:31.877082 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:31.878249 25665 webserver.cc:460] Webserver started at http://127.25.16.68:36661/ using document root <none> and password file <none>
I20240502 21:27:31.878443 25665 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:31.878510 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:31.878625 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:31.879190 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-3-root/instance:
uuid: "e8c1b0111c1f44ce906acee8fdfcec0e"
format_stamp: "Formatted at 2024-05-02 21:27:31 on dist-test-slave-b524"
I20240502 21:27:31.881561 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:31.883038 30606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:31.883358 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:31.883466 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-3-root
uuid: "e8c1b0111c1f44ce906acee8fdfcec0e"
format_stamp: "Formatted at 2024-05-02 21:27:31 on dist-test-slave-b524"
I20240502 21:27:31.883574 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task36zTzk/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714685126025081-25665-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:31.898218 25665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:31.898687 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:31.899204 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:31.900036 25665 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:31.900108 25665 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:31.900199 25665 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:31.900254 25665 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:31.912117 25665 rpc_server.cc:307] RPC server started. Bound to: 127.25.16.68:35903
I20240502 21:27:31.912143 30669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.16.68:35903 every 8 connection(s)
I20240502 21:27:31.918030 30670 heartbeater.cc:347] Connected to a master server at 127.25.16.126:39399
I20240502 21:27:31.918155 30670 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:31.918496 30670 heartbeater.cc:511] Master 127.25.16.126:39399 requested a full tablet report, sending...
I20240502 21:27:31.919227 30273 ts_manager.cc:194] Registered new tserver with Master: e8c1b0111c1f44ce906acee8fdfcec0e (127.25.16.68:35903)
I20240502 21:27:31.919955 30273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59748
I20240502 21:27:31.922132 30583 consensus_queue.cc:1035] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [LEADER]: Connected to new peer: Peer: permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:31.924430 30579 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240502 21:27:31.924525 30579 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240502 21:27:31.944057 30581 consensus_queue.cc:1035] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:31.949640 30583 consensus_queue.cc:1035] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:31.951200 30583 consensus_queue.cc:1035] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:31.956454 30581 consensus_queue.cc:1035] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:31.962793 30681 consensus_queue.cc:1035] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [LEADER]: Connected to new peer: Peer: permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:32.494607 30327 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:27:32.506076 30452 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3d136a54adf4ec99b4aa1104a714526"
dest_uuid: "926f90ca0bdc4abcb0a31f4ab996461d"
mode: GRACEFUL
new_leader_uuid: "ba0e34a385f748579ad428b1f9233390"
from {username='slave'} at 127.0.0.1:34184
I20240502 21:27:32.506320 30452 raft_consensus.cc:595] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 LEADER]: Received request to transfer leadership to TS ba0e34a385f748579ad428b1f9233390
I20240502 21:27:32.506748 30327 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 21:27:32.506685 30453 consensus_queue.cc:237] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } attrs { replace: true } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "e8c1b0111c1f44ce906acee8fdfcec0e" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 35903 } attrs { promote: true } }
I20240502 21:27:32.506891 30327 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:27:32.508592 30528 raft_consensus.cc:1261] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390 [term 1 FOLLOWER]: Refusing update from remote peer 926f90ca0bdc4abcb0a31f4ab996461d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.508816 30377 raft_consensus.cc:1261] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87 [term 1 FOLLOWER]: Refusing update from remote peer 926f90ca0bdc4abcb0a31f4ab996461d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.509078 30681 consensus_queue.cc:1035] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.509538 30583 consensus_queue.cc:1035] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [LEADER]: Connected to new peer: Peer: permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.511965 30528 raft_consensus.cc:2932] T a581356fbaf641258d2a92a451eaf30d P ba0e34a385f748579ad428b1f9233390 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e8c1b0111c1f44ce906acee8fdfcec0e (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } attrs { replace: true } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "e8c1b0111c1f44ce906acee8fdfcec0e" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 35903 } attrs { promote: true } } }
I20240502 21:27:32.511943 30377 raft_consensus.cc:2932] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e8c1b0111c1f44ce906acee8fdfcec0e (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } attrs { replace: true } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "e8c1b0111c1f44ce906acee8fdfcec0e" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 35903 } attrs { promote: true } } }
W20240502 21:27:32.515455 30417 consensus_peers.cc:485] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d -> Peer e8c1b0111c1f44ce906acee8fdfcec0e (127.25.16.68:35903): Couldn't send request to peer e8c1b0111c1f44ce906acee8fdfcec0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a581356fbaf641258d2a92a451eaf30d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.516384 30274 catalog_manager.cc:5484] T a581356fbaf641258d2a92a451eaf30d P 039e05b194a046cca2342c6efe73ab87 reported cstate change: config changed from index -1 to 2, NON_VOTER e8c1b0111c1f44ce906acee8fdfcec0e (127.25.16.68) added. New cstate: current_term: 1 leader_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } attrs { replace: true } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "e8c1b0111c1f44ce906acee8fdfcec0e" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 35903 } attrs { promote: true } } }
I20240502 21:27:32.557246 30681 raft_consensus.cc:2932] T a581356fbaf641258d2a92a451eaf30d P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e8c1b0111c1f44ce906acee8fdfcec0e (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } attrs { replace: true } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "e8c1b0111c1f44ce906acee8fdfcec0e" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 35903 } attrs { promote: true } } }
I20240502 21:27:32.566125 30377 consensus_queue.cc:237] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } attrs { replace: true } } peers { permanent_uuid: "e8c1b0111c1f44ce906acee8fdfcec0e" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 35903 } attrs { promote: true } }
I20240502 21:27:32.568032 30453 raft_consensus.cc:1261] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 FOLLOWER]: Refusing update from remote peer 039e05b194a046cca2342c6efe73ab87: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.568341 30528 raft_consensus.cc:1261] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390 [term 1 FOLLOWER]: Refusing update from remote peer 039e05b194a046cca2342c6efe73ab87: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.568578 30697 consensus_queue.cc:1035] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.568857 30684 consensus_queue.cc:1035] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:32.571625 30528 raft_consensus.cc:2932] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e8c1b0111c1f44ce906acee8fdfcec0e (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } attrs { replace: true } } peers { permanent_uuid: "e8c1b0111c1f44ce906acee8fdfcec0e" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 35903 } attrs { promote: true } } }
I20240502 21:27:32.571724 30453 raft_consensus.cc:2932] T ed648ab2b60944899b549bebb7bc602a P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e8c1b0111c1f44ce906acee8fdfcec0e (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } attrs { replace: true } } peers { permanent_uuid: "e8c1b0111c1f44ce906acee8fdfcec0e" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 35903 } attrs { promote: true } } }
W20240502 21:27:32.575603 30341 consensus_peers.cc:485] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 -> Peer e8c1b0111c1f44ce906acee8fdfcec0e (127.25.16.68:35903): Couldn't send request to peer e8c1b0111c1f44ce906acee8fdfcec0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed648ab2b60944899b549bebb7bc602a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.595690 30272 catalog_manager.cc:5484] T ed648ab2b60944899b549bebb7bc602a P ba0e34a385f748579ad428b1f9233390 reported cstate change: config changed from index -1 to 2, NON_VOTER e8c1b0111c1f44ce906acee8fdfcec0e (127.25.16.68) added. New cstate: current_term: 1 leader_uuid: "039e05b194a046cca2342c6efe73ab87" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } attrs { replace: true } } peers { permanent_uuid: "e8c1b0111c1f44ce906acee8fdfcec0e" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 35903 } attrs { promote: true } } }
I20240502 21:27:32.614599 30700 ts_tablet_manager.cc:902] T a581356fbaf641258d2a92a451eaf30d P e8c1b0111c1f44ce906acee8fdfcec0e: Initiating tablet copy from peer 926f90ca0bdc4abcb0a31f4ab996461d (127.25.16.66:36101)
F20240502 21:27:32.614907 30700 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1de7fda467344968ab81f653b8501210.queue_time_us, tablet-download-pool-1de7fda467344968ab81f653b8501210.queue_time_us) (tablet-download-pool-1de7fda467344968ab81f653b8501210.run_cpu_time_us, tablet-download-pool-1de7fda467344968ab81f653b8501210.run_cpu_time_us) (tablet-download-pool-1de7fda467344968ab81f653b8501210.run_wall_time_us, tablet-download-pool-1de7fda467344968ab81f653b8501210.run_wall_time_us) (tablet-download-pool-31c63a7053fa45b69eba6d7f7e78139b.queue_time_us, tablet-download-pool-31c63a7053fa45b69eba6d7f7e78139b.queue_time_us) (tablet-download-pool-31c63a7053fa45b69eba6d7f7e78139b.run_cpu_time_us, tablet-download-pool-31c63a7053fa45b69eba6d7f7e78139b.run_cpu_time_us) (tablet-download-pool-31c63a7053fa45b69eba6d7f7e78139b.run_wall_time_us, tablet-download-pool-31c63a7053fa45b69eba6d7f7e78139b.run_wall_time_us) (tablet-download-pool-5010a09a5243452dad7cd9b90535a229.queue_time_us, tablet-download-pool-5010a09a5243452dad7cd9b90535a229.queue_time_us) (tablet-download-pool-5010a09a5243452dad7cd9b90535a229.run_cpu_time_us, tablet-download-pool-5010a09a5243452dad7cd9b90535a229.run_cpu_time_us) (tablet-download-pool-5010a09a5243452dad7cd9b90535a229.run_wall_time_us, tablet-download-pool-5010a09a5243452dad7cd9b90535a229.run_wall_time_us) (tablet-download-pool-54a27d02080d420382add870508aaad4.queue_time_us, tablet-download-pool-54a27d02080d420382add870508aaad4.queue_time_us) (tablet-download-pool-54a27d02080d420382add870508aaad4.run_cpu_time_us, tablet-download-pool-54a27d02080d420382add870508aaad4.run_cpu_time_us) (tablet-download-pool-54a27d02080d420382add870508aaad4.run_wall_time_us, tablet-download-pool-54a27d02080d420382add870508aaad4.run_wall_time_us) (tablet-download-pool-6d08f3643db34c8ba4d4391b630be366.queue_time_us, tablet-download-pool-6d08f3643db34c8ba4d4391b630be366.queue_time_us) (tablet-download-pool-6d08f3643db34c8ba4d4391b630be366.run_cpu_time_us, tablet-download-pool-6d08f3643db34c8ba4d4391b630be366.run_cpu_time_us) (tablet-download-pool-6d08f3643db34c8ba4d4391b630be366.run_wall_time_us, tablet-download-pool-6d08f3643db34c8ba4d4391b630be366.run_wall_time_us) (tablet-download-pool-7ce57dd3a92645839765ba1ce6609430.queue_time_us, tablet-download-pool-7ce57dd3a92645839765ba1ce6609430.queue_time_us) (tablet-download-pool-7ce57dd3a92645839765ba1ce6609430.run_cpu_time_us, tablet-download-pool-7ce57dd3a92645839765ba1ce6609430.run_cpu_time_us) (tablet-download-pool-7ce57dd3a92645839765ba1ce6609430.run_wall_time_us, tablet-download-pool-7ce57dd3a92645839765ba1ce6609430.run_wall_time_us) (tablet-download-pool-a581356fbaf641258d2a92a451eaf30d.queue_time_us, tablet-download-pool-a581356fbaf641258d2a92a451eaf30d.queue_time_us) (tablet-download-pool-ab35657135e44445aee4aa6961f70c27.queue_time_us, tablet-download-pool-ab35657135e44445aee4aa6961f70c27.queue_time_us) (tablet-download-pool-ab35657135e44445aee4aa6961f70c27.run_cpu_time_us, tablet-download-pool-ab35657135e44445aee4aa6961f70c27.run_cpu_time_us) (tablet-download-pool-ab35657135e44445aee4aa6961f70c27.run_wall_time_us, tablet-download-pool-ab35657135e44445aee4aa6961f70c27.run_wall_time_us) (tablet-download-pool-af41f02157c94f55b2bc5090f566592e.queue_time_us, tablet-download-pool-af41f02157c94f55b2bc5090f566592e.queue_time_us) (tablet-download-pool-af41f02157c94f55b2bc5090f566592e.run_cpu_time_us, tablet-download-pool-af41f02157c94f55b2bc5090f566592e.run_cpu_time_us) (tablet-download-pool-af41f02157c94f55b2bc5090f566592e.run_wall_time_us, tablet-download-pool-af41f02157c94f55b2bc5090f566592e.run_wall_time_us) (tablet-download-pool-bdbac986d7b441818fc3b3038b76bb3e.queue_time_us, tablet-download-pool-bdbac986d7b441818fc3b3038b76bb3e.queue_time_us) (tablet-download-pool-bdbac986d7b441818fc3b3038b76bb3e.run_cpu_time_us, tablet-download-pool-bdbac986d7b441818fc3b3038b76bb3e.run_cpu_time_us) (tablet-download-pool-bdbac986d7b441818fc3b3038b76bb3e.run_wall_time_us, tablet-download-pool-bdbac986d7b441818fc3b3038b76bb3e.run_wall_time_us) (tablet-download-pool-ccd9ffc92c5c47d088c530e919f2ff77.queue_time_us, tablet-download-pool-ccd9ffc92c5c47d088c530e919f2ff77.queue_time_us) (tablet-download-pool-ccd9ffc92c5c47d088c530e919f2ff77.run_cpu_time_us, tablet-download-pool-ccd9ffc92c5c47d088c530e919f2ff77.run_cpu_time_us) (tablet-download-pool-ccd9ffc92c5c47d088c530e919f2ff77.run_wall_time_us, tablet-download-pool-ccd9ffc92c5c47d088c530e919f2ff77.run_wall_time_us) (tablet-download-pool-ea5a9bdae3f3459f915eaf3bdd3d346c.queue_time_us, tablet-download-pool-ea5a9bdae3f3459f915eaf3bdd3d346c.queue_time_us) (tablet-download-pool-ea5a9bdae3f3459f915eaf3bdd3d346c.run_cpu_time_us, tablet-download-pool-ea5a9bdae3f3459f915eaf3bdd3d346c.run_cpu_time_us) (tablet-download-pool-ea5a9bdae3f3459f915eaf3bdd3d346c.run_wall_time_us, tablet-download-pool-ea5a9bdae3f3459f915eaf3bdd3d346c.run_wall_time_us) (tablet-download-pool-ee4eba5acb8d4af986a00f937de0270e.queue_time_us, tablet-download-pool-ee4eba5acb8d4af986a00f937de0270e.queue_time_us) (tablet-download-pool-ee4eba5acb8d4af986a00f937de0270e.run_cpu_time_us, tablet-download-pool-ee4eba5acb8d4af986a00f937de0270e.run_cpu_time_us) (tablet-download-pool-ee4eba5acb8d4af986a00f937de0270e.run_wall_time_us, tablet-download-pool-ee4eba5acb8d4af986a00f937de0270e.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714685252 (unix time) try "date -d @1714685252" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800006441) received by PID 25665 (TID 0x7f3810d1a700) from PID 25665; stack trace: ***
@ 0x7f386d366980 (unknown) at ??:0
@ 0x7f38626b3fb7 gsignal at ??:0
@ 0x7f38626b5921 abort at ??:0
@ 0x7f3865168dcd google::LogMessage::Fail() at ??:0
I20240502 21:27:32.632529 30695 raft_consensus.cc:979] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d: : Instructing follower ba0e34a385f748579ad428b1f9233390 to start an election
I20240502 21:27:32.632700 30681 raft_consensus.cc:1067] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 LEADER]: Signalling peer ba0e34a385f748579ad428b1f9233390 to start an election
@ 0x7f386516cb93 google::LogMessage::SendToLog() at ??:0
I20240502 21:27:32.633311 30528 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3d136a54adf4ec99b4aa1104a714526"
dest_uuid: "ba0e34a385f748579ad428b1f9233390"
from {username='slave'} at 127.0.0.1:49928
I20240502 21:27:32.633517 30528 raft_consensus.cc:482] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:32.633628 30528 raft_consensus.cc:3037] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [term 1 FOLLOWER]: Advancing to term 2
@ 0x7f38651687cc google::LogMessage::Flush() at ??:0
I20240502 21:27:32.636436 30528 raft_consensus.cc:504] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:32.637274 30528 leader_election.cc:288] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [CANDIDATE]: Term 2 election: Requested vote from peers 039e05b194a046cca2342c6efe73ab87 (127.25.16.65:45931), 926f90ca0bdc4abcb0a31f4ab996461d (127.25.16.66:36101)
@ 0x7f3865169f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f38667e5742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f38667b1f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 21:27:32.650815 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d136a54adf4ec99b4aa1104a714526" candidate_uuid: "ba0e34a385f748579ad428b1f9233390" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "039e05b194a046cca2342c6efe73ab87"
I20240502 21:27:32.651021 30377 raft_consensus.cc:3037] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:32.651077 30453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d136a54adf4ec99b4aa1104a714526" candidate_uuid: "ba0e34a385f748579ad428b1f9233390" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "926f90ca0bdc4abcb0a31f4ab996461d"
I20240502 21:27:32.651297 30453 raft_consensus.cc:3032] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:32.651381 30453 raft_consensus.cc:729] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 926f90ca0bdc4abcb0a31f4ab996461d, State: Running, Role: LEADER
I20240502 21:27:32.651638 30453 consensus_queue.cc:260] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
I20240502 21:27:32.652020 30453 raft_consensus.cc:3037] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 1 FOLLOWER]: Advancing to term 2
@ 0x7f38667abcd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 21:27:32.653880 30377 raft_consensus.cc:2454] T c3d136a54adf4ec99b4aa1104a714526 P 039e05b194a046cca2342c6efe73ab87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0e34a385f748579ad428b1f9233390 in term 2.
I20240502 21:27:32.654822 30453 raft_consensus.cc:2454] T c3d136a54adf4ec99b4aa1104a714526 P 926f90ca0bdc4abcb0a31f4ab996461d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0e34a385f748579ad428b1f9233390 in term 2.
I20240502 21:27:32.655350 30492 leader_election.cc:302] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039e05b194a046cca2342c6efe73ab87, ba0e34a385f748579ad428b1f9233390; no voters:
I20240502 21:27:32.655649 30692 raft_consensus.cc:2781] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:32.656244 30692 raft_consensus.cc:686] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [term 2 LEADER]: Becoming Leader. State: Replica: ba0e34a385f748579ad428b1f9233390, State: Running, Role: LEADER
I20240502 21:27:32.656555 30692 consensus_queue.cc:237] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } }
@ 0x7f3875c778de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 21:27:32.658944 30273 catalog_manager.cc:5484] T c3d136a54adf4ec99b4aa1104a714526 P ba0e34a385f748579ad428b1f9233390 reported cstate change: term changed from 1 to 2, leader changed from 926f90ca0bdc4abcb0a31f4ab996461d (127.25.16.66) to ba0e34a385f748579ad428b1f9233390 (127.25.16.67). New cstate: current_term: 2 leader_uuid: "ba0e34a385f748579ad428b1f9233390" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } health_report { overall_health: HEALTHY } } }
@ 0x7f3875c8fdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f3875dba480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 21:27:32.666213 30595 raft_consensus.cc:2932] T ed648ab2b60944899b549bebb7bc602a P 039e05b194a046cca2342c6efe73ab87 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e8c1b0111c1f44ce906acee8fdfcec0e (127.25.16.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "039e05b194a046cca2342c6efe73ab87" member_type: VOTER last_known_addr { host: "127.25.16.65" port: 45931 } } peers { permanent_uuid: "ba0e34a385f748579ad428b1f9233390" member_type: VOTER last_known_addr { host: "127.25.16.67" port: 33973 } } peers { permanent_uuid: "926f90ca0bdc4abcb0a31f4ab996461d" member_type: VOTER last_known_addr { host: "127.25.16.66" port: 36101 } attrs { replace: true } } peers { permanent_uuid: "e8c1b0111c1f44ce906acee8fdfcec0e" member_type: NON_VOTER last_known_addr { host: "127.25.16.68" port: 35903 } attrs { promote: true } } }
@ 0x7f3875df2a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f3875dd5981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f3875dd570e std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f387ac7798d std::function<>::operator()() at ??:0
I20240502 21:27:32.688661 30717 ts_tablet_manager.cc:902] T ed648ab2b60944899b549bebb7bc602a P e8c1b0111c1f44ce906acee8fdfcec0e: Initiating tablet copy from peer 039e05b194a046cca2342c6efe73ab87 (127.25.16.65:45931)
@ 0x7f38667b5b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f38667b919c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f38667b9019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f387ac7798d std::function<>::operator()() at ??:0
@ 0x7f386679401e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f386d35b6db start_thread at ??:0
@ 0x7f386279671f clone at ??:0