[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 20:08:04.994465 11118 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.219.190:40449,127.10.219.189:43275,127.10.219.188:42837
I20240501 20:08:04.996577 11118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:04.998031 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:05.011754 11124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:05.012393 11123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:05.012944 11126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.013830 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:05.014883 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:05.015091 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:05.015192 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594085015168 us; error 0 us; skew 500 ppm
I20240501 20:08:05.015583 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:05.019060 11118 webserver.cc:460] Webserver started at http://127.10.219.190:35555/ using document root <none> and password file <none>
I20240501 20:08:05.020944 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:05.021334 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:05.022540 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:05.027426 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-0-root/instance:
uuid: "97e2a1517e7c4180b4397bcd1edde81d"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-n6pz"
I20240501 20:08:05.034916 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20240501 20:08:05.039165 11131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.041049 11118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240501 20:08:05.041275 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-0-root
uuid: "97e2a1517e7c4180b4397bcd1edde81d"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-n6pz"
I20240501 20:08:05.041505 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:05.073621 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:05.074668 11118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:05.075018 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:05.099500 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.190:40449
I20240501 20:08:05.099537 11182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.190:40449 every 8 connection(s)
I20240501 20:08:05.102192 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:05.106241 11185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:05.106781 11186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.107030 11183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:08:05.107515 11188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.108325 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:05.108676 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:05.108750 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:05.108834 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594085108794 us; error 0 us; skew 500 ppm
I20240501 20:08:05.109021 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:05.110672 11118 webserver.cc:460] Webserver started at http://127.10.219.189:40585/ using document root <none> and password file <none>
I20240501 20:08:05.110970 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:05.111086 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:05.111228 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:05.111953 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-1-root/instance:
uuid: "46a10eac6e9744a9a18bc8dbcff20361"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-n6pz"
I20240501 20:08:05.115170 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 20:08:05.117318 11193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.117743 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240501 20:08:05.117866 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-1-root
uuid: "46a10eac6e9744a9a18bc8dbcff20361"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-n6pz"
I20240501 20:08:05.118024 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:05.119591 11183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:08:05.141665 11183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } has no permanent_uuid. Determining permanent_uuid...
W20240501 20:08:05.143215 11133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.219.189:43275: connect: Connection refused (error 111)
W20240501 20:08:05.145115 11183 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.10.219.189:43275: Network error: Client connection negotiation failed: client connection to 127.10.219.189:43275: connect: Connection refused (error 111)
I20240501 20:08:05.153625 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:05.154464 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:05.174360 11247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.189:43275 every 8 connection(s)
I20240501 20:08:05.174382 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.189:43275
I20240501 20:08:05.176676 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:08:05.176828 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:08:05.181767 11251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.181860 11248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } has no permanent_uuid. Determining permanent_uuid...
W20240501 20:08:05.182585 11250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.183583 11118 server_base.cc:1034] running on GCE node
W20240501 20:08:05.184123 11253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.184613 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:05.184696 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:05.184783 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594085184742 us; error 0 us; skew 500 ppm
I20240501 20:08:05.184953 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:05.186558 11118 webserver.cc:460] Webserver started at http://127.10.219.188:35467/ using document root <none> and password file <none>
I20240501 20:08:05.186856 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:05.186967 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:05.187112 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:05.187933 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-2-root/instance:
uuid: "0cd26aae316e484b821941dd9dba1d6c"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-n6pz"
I20240501 20:08:05.190080 11248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:08:05.191649 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 20:08:05.194496 11183 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } attempt: 1
I20240501 20:08:05.195338 11260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.195962 11118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240501 20:08:05.196115 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-2-root
uuid: "0cd26aae316e484b821941dd9dba1d6c"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-n6pz"
I20240501 20:08:05.196298 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:05.204634 11248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:08:05.207348 11183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } has no permanent_uuid. Determining permanent_uuid...
W20240501 20:08:05.207549 11248 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.10.219.188:42837: Network error: Client connection negotiation failed: client connection to 127.10.219.188:42837: connect: Connection refused (error 111)
W20240501 20:08:05.209059 11183 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.10.219.188:42837: Network error: Client connection negotiation failed: client connection to 127.10.219.188:42837: connect: Connection refused (error 111)
I20240501 20:08:05.209805 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:05.210467 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:05.228504 11313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.188:42837 every 8 connection(s)
I20240501 20:08:05.228631 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.188:42837
I20240501 20:08:05.229938 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:08:05.230536 11314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:05.235030 11314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:08:05.244062 11314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:08:05.250363 11314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:08:05.252364 11183 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } attempt: 1
I20240501 20:08:05.260099 11248 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } attempt: 1
I20240501 20:08:05.265591 11314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c: Bootstrap starting.
I20240501 20:08:05.265847 11183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d: Bootstrap starting.
I20240501 20:08:05.270104 11248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361: Bootstrap starting.
I20240501 20:08:05.271548 11183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:05.271548 11314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:05.272593 11248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:05.273964 11248 log.cc:824] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:05.278265 11248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361: No bootstrap required, opened a new log
I20240501 20:08:05.278334 11183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d: No bootstrap required, opened a new log
I20240501 20:08:05.278364 11314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c: No bootstrap required, opened a new log
I20240501 20:08:05.286047 11248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } }
I20240501 20:08:05.286046 11314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } }
I20240501 20:08:05.286046 11183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } }
I20240501 20:08:05.286659 11248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:05.286666 11314 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:05.286878 11183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:05.286955 11314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cd26aae316e484b821941dd9dba1d6c, State: Initialized, Role: FOLLOWER
I20240501 20:08:05.287009 11183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e2a1517e7c4180b4397bcd1edde81d, State: Initialized, Role: FOLLOWER
I20240501 20:08:05.286952 11248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46a10eac6e9744a9a18bc8dbcff20361, State: Initialized, Role: FOLLOWER
I20240501 20:08:05.288050 11248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } }
I20240501 20:08:05.288085 11183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } }
I20240501 20:08:05.288132 11314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } }
I20240501 20:08:05.289250 11322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } } }
I20240501 20:08:05.289759 11322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:05.290766 11323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } } }
I20240501 20:08:05.291335 11248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:05.291612 11323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:05.291333 11183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:05.291270 11324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } } }
I20240501 20:08:05.292740 11324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:05.291332 11314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [sys.catalog]: configured and running, proceeding with master startup.
W20240501 20:08:05.308812 11348 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:08:05.309231 11348 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:08:05.311909 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240501 20:08:05.312089 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240501 20:08:05.313195 11357 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240501 20:08:05.313338 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:05.313340 11357 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 20:08:05.314137 11358 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:08:05.314272 11358 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 20:08:05.318750 11359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:05.319252 11360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:05.319978 11362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.320569 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:05.320993 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:05.321110 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:05.321183 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594085321157 us; error 0 us; skew 500 ppm
I20240501 20:08:05.321430 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:05.323347 11118 webserver.cc:460] Webserver started at http://127.10.219.129:42183/ using document root <none> and password file <none>
I20240501 20:08:05.323792 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:05.323933 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:05.324143 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:05.325295 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-0-root/instance:
uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-n6pz"
W20240501 20:08:05.328012 11137 tablet.cc:2369] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:05.329900 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240501 20:08:05.332906 11367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.333424 11118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:05.333549 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-0-root
uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-n6pz"
I20240501 20:08:05.333730 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:05.345017 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:05.345844 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:05.348030 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:05.349920 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:05.350087 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.350255 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:05.350371 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.374920 11429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.129:34251 every 8 connection(s)
I20240501 20:08:05.374904 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.129:34251
I20240501 20:08:05.403018 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:08:05.409839 11431 heartbeater.cc:347] Connected to a master server at 127.10.219.190:40449
I20240501 20:08:05.410367 11431 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:05.412015 11432 heartbeater.cc:347] Connected to a master server at 127.10.219.189:43275
I20240501 20:08:05.412241 11432 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:05.412333 11431 heartbeater.cc:511] Master 127.10.219.190:40449 requested a full tablet report, sending...
W20240501 20:08:05.413257 11437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.414094 11432 heartbeater.cc:511] Master 127.10.219.189:43275 requested a full tablet report, sending...
I20240501 20:08:05.416170 11148 ts_manager.cc:194] Registered new tserver with Master: cedf00ead5cd4ef0aae6dc9f7454dc44 (127.10.219.129:34251)
I20240501 20:08:05.416378 11213 ts_manager.cc:194] Registered new tserver with Master: cedf00ead5cd4ef0aae6dc9f7454dc44 (127.10.219.129:34251)
I20240501 20:08:05.417223 11430 heartbeater.cc:347] Connected to a master server at 127.10.219.188:42837
I20240501 20:08:05.417404 11430 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:05.419092 11430 heartbeater.cc:511] Master 127.10.219.188:42837 requested a full tablet report, sending...
I20240501 20:08:05.421126 11279 ts_manager.cc:194] Registered new tserver with Master: cedf00ead5cd4ef0aae6dc9f7454dc44 (127.10.219.129:34251)
W20240501 20:08:05.421389 11438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.428843 11118 server_base.cc:1034] running on GCE node
W20240501 20:08:05.429376 11440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.429855 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:05.429942 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:05.430038 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594085429986 us; error 0 us; skew 500 ppm
I20240501 20:08:05.430213 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:05.432070 11118 webserver.cc:460] Webserver started at http://127.10.219.130:33579/ using document root <none> and password file <none>
I20240501 20:08:05.432355 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:05.432469 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:05.432621 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:05.433423 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-1-root/instance:
uuid: "c2f390acff5140698be53aef35ec8ba3"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-n6pz"
I20240501 20:08:05.436842 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:05.439076 11445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.439579 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 20:08:05.439760 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-1-root
uuid: "c2f390acff5140698be53aef35ec8ba3"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-n6pz"
I20240501 20:08:05.439965 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:05.480501 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:05.481375 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:05.482239 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:05.483546 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:05.483641 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.483777 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:05.483844 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.508777 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.130:36223
I20240501 20:08:05.508801 11507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.130:36223 every 8 connection(s)
I20240501 20:08:05.524608 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:08:05.535887 11508 heartbeater.cc:347] Connected to a master server at 127.10.219.188:42837
I20240501 20:08:05.536134 11508 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:05.537076 11508 heartbeater.cc:511] Master 127.10.219.188:42837 requested a full tablet report, sending...
I20240501 20:08:05.539369 11510 heartbeater.cc:347] Connected to a master server at 127.10.219.189:43275
I20240501 20:08:05.539860 11509 heartbeater.cc:347] Connected to a master server at 127.10.219.190:40449
I20240501 20:08:05.540375 11510 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:05.540464 11279 ts_manager.cc:194] Registered new tserver with Master: c2f390acff5140698be53aef35ec8ba3 (127.10.219.130:36223)
W20240501 20:08:05.540858 11515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:05.540910 11516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.541472 11509 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:05.542670 11509 heartbeater.cc:511] Master 127.10.219.190:40449 requested a full tablet report, sending...
I20240501 20:08:05.544389 11148 ts_manager.cc:194] Registered new tserver with Master: c2f390acff5140698be53aef35ec8ba3 (127.10.219.130:36223)
I20240501 20:08:05.545032 11510 heartbeater.cc:511] Master 127.10.219.189:43275 requested a full tablet report, sending...
I20240501 20:08:05.546339 11213 ts_manager.cc:194] Registered new tserver with Master: c2f390acff5140698be53aef35ec8ba3 (127.10.219.130:36223)
W20240501 20:08:05.547936 11518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:05.548648 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:05.549005 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:05.549108 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:05.549188 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594085549159 us; error 0 us; skew 500 ppm
I20240501 20:08:05.549397 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:05.551285 11118 webserver.cc:460] Webserver started at http://127.10.219.131:37629/ using document root <none> and password file <none>
I20240501 20:08:05.551662 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:05.551772 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:05.551960 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:05.552922 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-2-root/instance:
uuid: "7d1342094fe2492299589985d090cf2c"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-n6pz"
I20240501 20:08:05.556902 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 20:08:05.559470 11523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.560220 11118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:08:05.560402 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-2-root
uuid: "7d1342094fe2492299589985d090cf2c"
format_stamp: "Formatted at 2024-05-01 20:08:05 on dist-test-slave-n6pz"
I20240501 20:08:05.560564 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:05.580490 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:05.581475 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:05.582356 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:05.583729 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:05.583858 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.583974 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:05.584061 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:05.612632 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.131:41041
I20240501 20:08:05.612660 11585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.131:41041 every 8 connection(s)
I20240501 20:08:05.625244 11322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:05.625721 11322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } }
I20240501 20:08:05.626994 11587 heartbeater.cc:347] Connected to a master server at 127.10.219.190:40449
I20240501 20:08:05.627197 11587 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:05.627756 11587 heartbeater.cc:511] Master 127.10.219.190:40449 requested a full tablet report, sending...
I20240501 20:08:05.629124 11148 ts_manager.cc:194] Registered new tserver with Master: 7d1342094fe2492299589985d090cf2c (127.10.219.131:41041)
I20240501 20:08:05.629565 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46a10eac6e9744a9a18bc8dbcff20361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e2a1517e7c4180b4397bcd1edde81d" is_pre_election: true
I20240501 20:08:05.630352 11158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46a10eac6e9744a9a18bc8dbcff20361 in term 0.
I20240501 20:08:05.631244 11194 leader_election.cc:302] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46a10eac6e9744a9a18bc8dbcff20361, 97e2a1517e7c4180b4397bcd1edde81d; no voters:
I20240501 20:08:05.632488 11591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:05.633008 11322 leader_election.cc:288] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97e2a1517e7c4180b4397bcd1edde81d (127.10.219.190:40449), 0cd26aae316e484b821941dd9dba1d6c (127.10.219.188:42837)
I20240501 20:08:05.633127 11289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46a10eac6e9744a9a18bc8dbcff20361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cd26aae316e484b821941dd9dba1d6c" is_pre_election: true
I20240501 20:08:05.633239 11591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:05.633478 11289 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46a10eac6e9744a9a18bc8dbcff20361 in term 0.
I20240501 20:08:05.633668 11591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:05.636166 11586 heartbeater.cc:347] Connected to a master server at 127.10.219.188:42837
I20240501 20:08:05.636351 11586 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:05.637008 11586 heartbeater.cc:511] Master 127.10.219.188:42837 requested a full tablet report, sending...
I20240501 20:08:05.638178 11279 ts_manager.cc:194] Registered new tserver with Master: 7d1342094fe2492299589985d090cf2c (127.10.219.131:41041)
I20240501 20:08:05.645992 11591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } }
I20240501 20:08:05.647024 11591 leader_election.cc:288] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [CANDIDATE]: Term 1 election: Requested vote from peers 97e2a1517e7c4180b4397bcd1edde81d (127.10.219.190:40449), 0cd26aae316e484b821941dd9dba1d6c (127.10.219.188:42837)
I20240501 20:08:05.647424 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46a10eac6e9744a9a18bc8dbcff20361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e2a1517e7c4180b4397bcd1edde81d"
I20240501 20:08:05.647528 11289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46a10eac6e9744a9a18bc8dbcff20361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cd26aae316e484b821941dd9dba1d6c"
I20240501 20:08:05.647650 11158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:05.647753 11289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:05.651145 11158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46a10eac6e9744a9a18bc8dbcff20361 in term 1.
I20240501 20:08:05.651417 11289 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46a10eac6e9744a9a18bc8dbcff20361 in term 1.
I20240501 20:08:05.651743 11588 heartbeater.cc:347] Connected to a master server at 127.10.219.189:43275
I20240501 20:08:05.651803 11194 leader_election.cc:302] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46a10eac6e9744a9a18bc8dbcff20361, 97e2a1517e7c4180b4397bcd1edde81d; no voters:
I20240501 20:08:05.651966 11588 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:05.652259 11591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:05.652500 11591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 1 LEADER]: Becoming Leader. State: Replica: 46a10eac6e9744a9a18bc8dbcff20361, State: Running, Role: LEADER
I20240501 20:08:05.652752 11588 heartbeater.cc:511] Master 127.10.219.189:43275 requested a full tablet report, sending...
I20240501 20:08:05.653240 11591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } }
I20240501 20:08:05.658048 11213 ts_manager.cc:194] Registered new tserver with Master: 7d1342094fe2492299589985d090cf2c (127.10.219.131:41041)
I20240501 20:08:05.658699 11118 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015694897s
I20240501 20:08:05.658756 11322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46a10eac6e9744a9a18bc8dbcff20361. Latest consensus state: current_term: 1 leader_uuid: "46a10eac6e9744a9a18bc8dbcff20361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } } }
I20240501 20:08:05.659080 11322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:05.660058 11594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:05.662429 11594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:05.671329 11289 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [term 1 FOLLOWER]: Refusing update from remote peer 46a10eac6e9744a9a18bc8dbcff20361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:05.671327 11158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [term 1 FOLLOWER]: Refusing update from remote peer 46a10eac6e9744a9a18bc8dbcff20361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:05.672358 11322 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:05.673274 11591 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:05.679128 11323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46a10eac6e9744a9a18bc8dbcff20361. Latest consensus state: current_term: 1 leader_uuid: "46a10eac6e9744a9a18bc8dbcff20361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } } }
I20240501 20:08:05.679612 11323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:05.683305 11591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46a10eac6e9744a9a18bc8dbcff20361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } } }
I20240501 20:08:05.683697 11591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:05.687206 11594 catalog_manager.cc:1347] Generated new cluster ID: 6bcb5b6a14e24584ae0fe3bc4875d464
I20240501 20:08:05.687412 11594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:05.694622 11323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46a10eac6e9744a9a18bc8dbcff20361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } } }
I20240501 20:08:05.695015 11323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:05.695294 11324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46a10eac6e9744a9a18bc8dbcff20361. Latest consensus state: current_term: 1 leader_uuid: "46a10eac6e9744a9a18bc8dbcff20361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } } }
I20240501 20:08:05.695642 11324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:05.697126 11322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46a10eac6e9744a9a18bc8dbcff20361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } } }
I20240501 20:08:05.697399 11322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:05.700160 11597 mvcc.cc:205] Tried to move back new op lower bound from 7022977374897971200 to 7022977374841958400. Current Snapshot: MvccSnapshot[applied={T|T < 7022977374897971200}]
I20240501 20:08:05.701555 11324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46a10eac6e9744a9a18bc8dbcff20361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e2a1517e7c4180b4397bcd1edde81d" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 40449 } } peers { permanent_uuid: "46a10eac6e9744a9a18bc8dbcff20361" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 43275 } } peers { permanent_uuid: "0cd26aae316e484b821941dd9dba1d6c" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 42837 } } }
I20240501 20:08:05.701843 11324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:05.720341 11594 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:05.722128 11594 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:05.736735 11594 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361: Generated new TSK 0
I20240501 20:08:05.738605 11594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:06.311470 11348 catalog_manager.cc:1259] Loaded cluster ID: 6bcb5b6a14e24584ae0fe3bc4875d464
I20240501 20:08:06.311618 11348 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c: loading cluster ID for follower catalog manager: success
I20240501 20:08:06.313925 11348 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c: acquiring CA information for follower catalog manager: success
I20240501 20:08:06.315853 11358 catalog_manager.cc:1259] Loaded cluster ID: 6bcb5b6a14e24584ae0fe3bc4875d464
I20240501 20:08:06.315930 11348 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 20:08:06.315989 11358 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d: loading cluster ID for follower catalog manager: success
I20240501 20:08:06.317867 11358 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d: acquiring CA information for follower catalog manager: success
I20240501 20:08:06.318986 11358 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 20:08:06.421805 11213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42136
I20240501 20:08:06.548769 11213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42138
I20240501 20:08:06.661029 11213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42140
I20240501 20:08:07.311043 11355 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:07.424077 11432 heartbeater.cc:503] Master 127.10.219.189:43275 was elected leader, sending a full tablet report...
I20240501 20:08:07.550175 11510 heartbeater.cc:503] Master 127.10.219.189:43275 was elected leader, sending a full tablet report...
I20240501 20:08:07.662515 11588 heartbeater.cc:503] Master 127.10.219.189:43275 was elected leader, sending a full tablet report...
I20240501 20:08:07.823413 11118 test_util.cc:268] Using random seed: 193587221
I20240501 20:08:07.860710 11213 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49932:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 20:08:07.921973 11473 tablet_service.cc:1467] Processing CreateTablet for tablet d64c141c452e42ce9e8f7c23251477a7 (DEFAULT_TABLE table=test-workload [id=424063ac1eb34d0a8a440717fa93e05c]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:08:07.923573 11473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64c141c452e42ce9e8f7c23251477a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:07.928563 11551 tablet_service.cc:1467] Processing CreateTablet for tablet 966f1fc09aa54767970c045cd55e952e (DEFAULT_TABLE table=test-workload [id=424063ac1eb34d0a8a440717fa93e05c]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:08:07.929426 11551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966f1fc09aa54767970c045cd55e952e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:07.932018 11395 tablet_service.cc:1467] Processing CreateTablet for tablet 43ab2ad5fcb349deae7afedeba3e7719 (DEFAULT_TABLE table=test-workload [id=424063ac1eb34d0a8a440717fa93e05c]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:08:07.932806 11395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ab2ad5fcb349deae7afedeba3e7719. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:07.934051 11633 tablet_bootstrap.cc:492] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3: Bootstrap starting.
I20240501 20:08:07.935215 11394 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd30dbfe5bb4e019ef48f7d9f6dc680 (DEFAULT_TABLE table=test-workload [id=424063ac1eb34d0a8a440717fa93e05c]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:08:07.936442 11394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd30dbfe5bb4e019ef48f7d9f6dc680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:07.939064 11633 tablet_bootstrap.cc:654] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:07.951182 11633 tablet_bootstrap.cc:492] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3: No bootstrap required, opened a new log
I20240501 20:08:07.951473 11633 ts_tablet_manager.cc:1367] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.010s
I20240501 20:08:07.952527 11633 raft_consensus.cc:348] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f390acff5140698be53aef35ec8ba3" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36223 } }
I20240501 20:08:07.952761 11633 raft_consensus.cc:374] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:07.952898 11633 raft_consensus.cc:729] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f390acff5140698be53aef35ec8ba3, State: Initialized, Role: FOLLOWER
I20240501 20:08:07.953186 11633 consensus_queue.cc:260] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f390acff5140698be53aef35ec8ba3" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36223 } }
I20240501 20:08:07.953423 11633 raft_consensus.cc:388] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:07.953529 11633 raft_consensus.cc:482] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:07.953656 11633 raft_consensus.cc:3037] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:07.955017 11636 tablet_bootstrap.cc:492] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44: Bootstrap starting.
I20240501 20:08:07.956351 11634 tablet_bootstrap.cc:492] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c: Bootstrap starting.
I20240501 20:08:07.957397 11633 raft_consensus.cc:504] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f390acff5140698be53aef35ec8ba3" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36223 } }
I20240501 20:08:07.957890 11633 leader_election.cc:302] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2f390acff5140698be53aef35ec8ba3; no voters:
I20240501 20:08:07.959247 11633 leader_election.cc:288] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:07.959769 11634 tablet_bootstrap.cc:654] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:07.959807 11637 raft_consensus.cc:2781] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:07.959785 11636 tablet_bootstrap.cc:654] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:07.962409 11633 ts_tablet_manager.cc:1398] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20240501 20:08:07.962953 11637 raft_consensus.cc:686] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [term 1 LEADER]: Becoming Leader. State: Replica: c2f390acff5140698be53aef35ec8ba3, State: Running, Role: LEADER
I20240501 20:08:07.963680 11637 consensus_queue.cc:237] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f390acff5140698be53aef35ec8ba3" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36223 } }
I20240501 20:08:07.966013 11636 tablet_bootstrap.cc:492] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44: No bootstrap required, opened a new log
I20240501 20:08:07.966233 11636 ts_tablet_manager.cc:1367] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 20:08:07.967149 11636 raft_consensus.cc:348] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 34251 } }
I20240501 20:08:07.967327 11636 raft_consensus.cc:374] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:07.967437 11636 raft_consensus.cc:729] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedf00ead5cd4ef0aae6dc9f7454dc44, State: Initialized, Role: FOLLOWER
I20240501 20:08:07.967698 11636 consensus_queue.cc:260] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 34251 } }
I20240501 20:08:07.967901 11636 raft_consensus.cc:388] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:07.967979 11636 raft_consensus.cc:482] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:07.968088 11636 raft_consensus.cc:3037] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:07.970844 11636 raft_consensus.cc:504] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 34251 } }
I20240501 20:08:07.971277 11636 leader_election.cc:302] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cedf00ead5cd4ef0aae6dc9f7454dc44; no voters:
I20240501 20:08:07.972728 11634 tablet_bootstrap.cc:492] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c: No bootstrap required, opened a new log
I20240501 20:08:07.973057 11634 ts_tablet_manager.cc:1367] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20240501 20:08:07.974349 11634 raft_consensus.cc:348] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1342094fe2492299589985d090cf2c" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 41041 } }
I20240501 20:08:07.974581 11636 leader_election.cc:288] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:07.974758 11634 raft_consensus.cc:374] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:07.974856 11644 raft_consensus.cc:2781] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:07.975044 11634 raft_consensus.cc:729] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1342094fe2492299589985d090cf2c, State: Initialized, Role: FOLLOWER
I20240501 20:08:07.977862 11213 catalog_manager.cc:5484] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2f390acff5140698be53aef35ec8ba3 (127.10.219.130). New cstate: current_term: 1 leader_uuid: "c2f390acff5140698be53aef35ec8ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f390acff5140698be53aef35ec8ba3" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36223 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:07.979857 11634 consensus_queue.cc:260] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1342094fe2492299589985d090cf2c" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 41041 } }
I20240501 20:08:07.980350 11634 raft_consensus.cc:388] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:07.980705 11634 raft_consensus.cc:482] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:07.981439 11634 raft_consensus.cc:3037] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:07.985190 11644 raft_consensus.cc:686] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 1 LEADER]: Becoming Leader. State: Replica: cedf00ead5cd4ef0aae6dc9f7454dc44, State: Running, Role: LEADER
I20240501 20:08:07.986449 11636 ts_tablet_manager.cc:1398] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44: Time spent starting tablet: real 0.020s user 0.006s sys 0.009s
I20240501 20:08:07.985991 11644 consensus_queue.cc:237] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 34251 } }
I20240501 20:08:07.988730 11634 raft_consensus.cc:504] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1342094fe2492299589985d090cf2c" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 41041 } }
I20240501 20:08:07.989157 11634 leader_election.cc:302] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d1342094fe2492299589985d090cf2c; no voters:
I20240501 20:08:07.990522 11636 tablet_bootstrap.cc:492] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44: Bootstrap starting.
I20240501 20:08:07.994040 11636 tablet_bootstrap.cc:654] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:07.996639 11634 leader_election.cc:288] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:07.998099 11648 raft_consensus.cc:2781] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:07.999781 11213 catalog_manager.cc:5484] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 reported cstate change: term changed from 0 to 1, leader changed from <none> to cedf00ead5cd4ef0aae6dc9f7454dc44 (127.10.219.129). New cstate: current_term: 1 leader_uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 34251 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:08.001415 11648 raft_consensus.cc:686] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [term 1 LEADER]: Becoming Leader. State: Replica: 7d1342094fe2492299589985d090cf2c, State: Running, Role: LEADER
I20240501 20:08:08.002018 11648 consensus_queue.cc:237] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1342094fe2492299589985d090cf2c" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 41041 } }
I20240501 20:08:08.017676 11212 catalog_manager.cc:5484] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d1342094fe2492299589985d090cf2c (127.10.219.131). New cstate: current_term: 1 leader_uuid: "7d1342094fe2492299589985d090cf2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1342094fe2492299589985d090cf2c" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 41041 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:08.049250 11634 ts_tablet_manager.cc:1398] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c: Time spent starting tablet: real 0.076s user 0.033s sys 0.021s
I20240501 20:08:08.055454 11636 tablet_bootstrap.cc:492] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44: No bootstrap required, opened a new log
I20240501 20:08:08.057202 11636 ts_tablet_manager.cc:1367] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44: Time spent bootstrapping tablet: real 0.069s user 0.022s sys 0.017s
I20240501 20:08:08.058358 11636 raft_consensus.cc:348] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 34251 } }
I20240501 20:08:08.058609 11636 raft_consensus.cc:374] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:08.058753 11636 raft_consensus.cc:729] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedf00ead5cd4ef0aae6dc9f7454dc44, State: Initialized, Role: FOLLOWER
I20240501 20:08:08.059069 11636 consensus_queue.cc:260] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 34251 } }
I20240501 20:08:08.059373 11636 raft_consensus.cc:388] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:08.059579 11636 raft_consensus.cc:482] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:08.059752 11636 raft_consensus.cc:3037] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:08.063565 11636 raft_consensus.cc:504] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 34251 } }
I20240501 20:08:08.063980 11636 leader_election.cc:302] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cedf00ead5cd4ef0aae6dc9f7454dc44; no voters:
I20240501 20:08:08.064292 11636 leader_election.cc:288] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:08.064312 11644 raft_consensus.cc:2781] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:08.064955 11636 ts_tablet_manager.cc:1398] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:08:08.065208 11644 raft_consensus.cc:686] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 1 LEADER]: Becoming Leader. State: Replica: cedf00ead5cd4ef0aae6dc9f7454dc44, State: Running, Role: LEADER
I20240501 20:08:08.065552 11644 consensus_queue.cc:237] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 34251 } }
I20240501 20:08:08.070410 11213 catalog_manager.cc:5484] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 reported cstate change: term changed from 0 to 1, leader changed from <none> to cedf00ead5cd4ef0aae6dc9f7454dc44 (127.10.219.129). New cstate: current_term: 1 leader_uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cedf00ead5cd4ef0aae6dc9f7454dc44" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 34251 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:08.109886 11118 tablet_server.cc:178] TabletServer@127.10.219.129:0 shutting down...
I20240501 20:08:08.121811 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:08.122280 11118 tablet_replica.cc:330] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44: stopping tablet replica
I20240501 20:08:08.122758 11118 raft_consensus.cc:2229] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:08.123036 11118 raft_consensus.cc:2258] T 9dd30dbfe5bb4e019ef48f7d9f6dc680 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:08.124838 11118 tablet_replica.cc:330] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44: stopping tablet replica
I20240501 20:08:08.124989 11118 raft_consensus.cc:2229] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:08.125149 11118 raft_consensus.cc:2258] T 43ab2ad5fcb349deae7afedeba3e7719 P cedf00ead5cd4ef0aae6dc9f7454dc44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:08.141440 11118 tablet_server.cc:194] TabletServer@127.10.219.129:0 shutdown complete.
I20240501 20:08:08.147711 11118 tablet_server.cc:178] TabletServer@127.10.219.130:0 shutting down...
I20240501 20:08:08.157197 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:08.157719 11118 tablet_replica.cc:330] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3: stopping tablet replica
I20240501 20:08:08.157940 11118 raft_consensus.cc:2229] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:08.158159 11118 raft_consensus.cc:2258] T d64c141c452e42ce9e8f7c23251477a7 P c2f390acff5140698be53aef35ec8ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:08.173126 11118 tablet_server.cc:194] TabletServer@127.10.219.130:0 shutdown complete.
I20240501 20:08:08.176541 11118 tablet_server.cc:178] TabletServer@127.10.219.131:0 shutting down...
I20240501 20:08:08.186226 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:08.186714 11118 tablet_replica.cc:330] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c: stopping tablet replica
I20240501 20:08:08.186944 11118 raft_consensus.cc:2229] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:08.187139 11118 raft_consensus.cc:2258] T 966f1fc09aa54767970c045cd55e952e P 7d1342094fe2492299589985d090cf2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:08.191856 11118 tablet_server.cc:194] TabletServer@127.10.219.131:0 shutdown complete.
I20240501 20:08:08.195297 11118 master.cc:537] Master@127.10.219.190:40449 shutting down...
I20240501 20:08:08.204373 11118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:08.204700 11118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:08.204874 11118 tablet_replica.cc:330] T 00000000000000000000000000000000 P 97e2a1517e7c4180b4397bcd1edde81d: stopping tablet replica
I20240501 20:08:08.219915 11118 master.cc:552] Master@127.10.219.190:40449 shutdown complete.
I20240501 20:08:08.225970 11118 master.cc:537] Master@127.10.219.189:43275 shutting down...
I20240501 20:08:08.234488 11118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:08.235042 11118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:08.235184 11118 tablet_replica.cc:330] T 00000000000000000000000000000000 P 46a10eac6e9744a9a18bc8dbcff20361: stopping tablet replica
I20240501 20:08:08.250561 11118 master.cc:552] Master@127.10.219.189:43275 shutdown complete.
I20240501 20:08:08.255650 11118 master.cc:537] Master@127.10.219.188:42837 shutting down...
I20240501 20:08:08.263922 11118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:08.264219 11118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:08.264374 11118 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0cd26aae316e484b821941dd9dba1d6c: stopping tablet replica
I20240501 20:08:08.278919 11118 master.cc:552] Master@127.10.219.188:42837 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3302 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240501 20:08:08.294242 11118 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.219.190:41825
I20240501 20:08:08.294883 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:08.298169 11656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:08.298835 11657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:08.299523 11659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:08.300246 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:08.300556 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:08.300633 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:08.300671 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594088300658 us; error 0 us; skew 500 ppm
I20240501 20:08:08.300833 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:08.308228 11118 webserver.cc:460] Webserver started at http://127.10.219.190:37161/ using document root <none> and password file <none>
I20240501 20:08:08.308557 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:08.308632 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:08.308775 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:08.309480 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/master-0-root/instance:
uuid: "54e9aabab6d845698159a20b235b874d"
format_stamp: "Formatted at 2024-05-01 20:08:08 on dist-test-slave-n6pz"
I20240501 20:08:08.312623 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:08.314584 11664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:08.315039 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 20:08:08.315161 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/master-0-root
uuid: "54e9aabab6d845698159a20b235b874d"
format_stamp: "Formatted at 2024-05-01 20:08:08 on dist-test-slave-n6pz"
I20240501 20:08:08.315308 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:08.333297 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:08.334017 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:08.350281 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.190:41825
I20240501 20:08:08.350294 11715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.190:41825 every 8 connection(s)
I20240501 20:08:08.351907 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:08.356352 11716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d: Bootstrap starting.
I20240501 20:08:08.357841 11716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:08.359702 11716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d: No bootstrap required, opened a new log
I20240501 20:08:08.360363 11716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e9aabab6d845698159a20b235b874d" member_type: VOTER }
I20240501 20:08:08.360525 11716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:08.360595 11716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e9aabab6d845698159a20b235b874d, State: Initialized, Role: FOLLOWER
I20240501 20:08:08.360785 11716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e9aabab6d845698159a20b235b874d" member_type: VOTER }
I20240501 20:08:08.360913 11716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:08.360975 11716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:08.361050 11716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:08.363111 11716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e9aabab6d845698159a20b235b874d" member_type: VOTER }
I20240501 20:08:08.363363 11716 leader_election.cc:302] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54e9aabab6d845698159a20b235b874d; no voters:
I20240501 20:08:08.363858 11716 leader_election.cc:288] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:08.364045 11719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:08.364607 11719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [term 1 LEADER]: Becoming Leader. State: Replica: 54e9aabab6d845698159a20b235b874d, State: Running, Role: LEADER
I20240501 20:08:08.364941 11716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:08.364961 11719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e9aabab6d845698159a20b235b874d" member_type: VOTER }
I20240501 20:08:08.366124 11720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54e9aabab6d845698159a20b235b874d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e9aabab6d845698159a20b235b874d" member_type: VOTER } }
I20240501 20:08:08.366200 11721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54e9aabab6d845698159a20b235b874d. Latest consensus state: current_term: 1 leader_uuid: "54e9aabab6d845698159a20b235b874d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e9aabab6d845698159a20b235b874d" member_type: VOTER } }
I20240501 20:08:08.366533 11721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:08.366851 11720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:08.368024 11724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:08.370755 11724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:08.374624 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:08:08.374910 11724 catalog_manager.cc:1347] Generated new cluster ID: b35862782c724b7eba51af293a67abc8
I20240501 20:08:08.375000 11724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:08.407052 11724 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:08.407958 11724 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:08.415781 11724 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d: Generated new TSK 0
I20240501 20:08:08.416124 11724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:08.439953 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:08.443778 11737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:08.444432 11738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:08.445180 11740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:08.445876 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:08.446182 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:08.446252 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:08.446338 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594088446277 us; error 0 us; skew 500 ppm
I20240501 20:08:08.446513 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:08.448009 11118 webserver.cc:460] Webserver started at http://127.10.219.129:45669/ using document root <none> and password file <none>
I20240501 20:08:08.448282 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:08.448382 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:08.448541 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:08.449280 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-0-root/instance:
uuid: "45dff34340a948dabf879205d1ea305c"
format_stamp: "Formatted at 2024-05-01 20:08:08 on dist-test-slave-n6pz"
I20240501 20:08:08.452179 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 20:08:08.454056 11745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:08.454568 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240501 20:08:08.454700 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-0-root
uuid: "45dff34340a948dabf879205d1ea305c"
format_stamp: "Formatted at 2024-05-01 20:08:08 on dist-test-slave-n6pz"
I20240501 20:08:08.454869 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:08.465484 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:08.466454 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:08.467235 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:08.468569 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:08.468712 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:08.468847 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:08.468905 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:08.487308 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.129:36653
I20240501 20:08:08.487347 11807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.129:36653 every 8 connection(s)
I20240501 20:08:08.489307 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:08.494663 11813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:08.494822 11812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:08.495968 11815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:08.496701 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:08.497002 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:08.497097 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:08.497174 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594088497145 us; error 0 us; skew 500 ppm
I20240501 20:08:08.497407 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:08.498070 11808 heartbeater.cc:347] Connected to a master server at 127.10.219.190:41825
I20240501 20:08:08.498250 11808 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:08.498781 11808 heartbeater.cc:511] Master 127.10.219.190:41825 requested a full tablet report, sending...
I20240501 20:08:08.498975 11118 webserver.cc:460] Webserver started at http://127.10.219.130:41831/ using document root <none> and password file <none>
I20240501 20:08:08.499286 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:08.499400 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:08.499586 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:08.499866 11681 ts_manager.cc:194] Registered new tserver with Master: 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653)
I20240501 20:08:08.500397 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-1-root/instance:
uuid: "9a2676a1687a4171bb53684816779348"
format_stamp: "Formatted at 2024-05-01 20:08:08 on dist-test-slave-n6pz"
I20240501 20:08:08.501101 11681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55094
I20240501 20:08:08.503237 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240501 20:08:08.504968 11820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:08.505379 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 20:08:08.505497 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-1-root
uuid: "9a2676a1687a4171bb53684816779348"
format_stamp: "Formatted at 2024-05-01 20:08:08 on dist-test-slave-n6pz"
I20240501 20:08:08.505614 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:08.524546 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:08.525197 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:08.525840 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:08.526829 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:08.526922 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:08.527037 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:08.527117 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:08.547894 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.130:41381
I20240501 20:08:08.547995 11882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.130:41381 every 8 connection(s)
I20240501 20:08:08.550477 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:08.555335 11886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:08.556398 11887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:08.560642 11883 heartbeater.cc:347] Connected to a master server at 127.10.219.190:41825
I20240501 20:08:08.560875 11883 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:08.561146 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:08.561533 11883 heartbeater.cc:511] Master 127.10.219.190:41825 requested a full tablet report, sending...
W20240501 20:08:08.561717 11889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:08.562173 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:08.562270 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:08.562399 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594088562345 us; error 0 us; skew 500 ppm
I20240501 20:08:08.562572 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:08.562868 11681 ts_manager.cc:194] Registered new tserver with Master: 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381)
I20240501 20:08:08.563858 11681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55102
I20240501 20:08:08.564420 11118 webserver.cc:460] Webserver started at http://127.10.219.131:38863/ using document root <none> and password file <none>
I20240501 20:08:08.564770 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:08.564939 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:08.565160 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:08.566013 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-2-root/instance:
uuid: "ceb448c0562d444e8cbf5f256b20e9bc"
format_stamp: "Formatted at 2024-05-01 20:08:08 on dist-test-slave-n6pz"
I20240501 20:08:08.569173 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 20:08:08.571610 11894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:08.572213 11118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:08:08.572382 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-2-root
uuid: "ceb448c0562d444e8cbf5f256b20e9bc"
format_stamp: "Formatted at 2024-05-01 20:08:08 on dist-test-slave-n6pz"
I20240501 20:08:08.572589 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:08.581921 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:08.582876 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:08.583775 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:08.585111 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:08.585216 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:08.585489 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:08.585587 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240501 20:08:08.606916 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.131:36485
I20240501 20:08:08.606940 11956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.131:36485 every 8 connection(s)
I20240501 20:08:08.614586 11957 heartbeater.cc:347] Connected to a master server at 127.10.219.190:41825
I20240501 20:08:08.614760 11957 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:08.615192 11957 heartbeater.cc:511] Master 127.10.219.190:41825 requested a full tablet report, sending...
I20240501 20:08:08.616230 11681 ts_manager.cc:194] Registered new tserver with Master: ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485)
I20240501 20:08:08.617028 11681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55110
I20240501 20:08:08.617283 11118 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.009297722s
I20240501 20:08:09.502597 11808 heartbeater.cc:503] Master 127.10.219.190:41825 was elected leader, sending a full tablet report...
I20240501 20:08:09.565634 11883 heartbeater.cc:503] Master 127.10.219.190:41825 was elected leader, sending a full tablet report...
I20240501 20:08:09.618530 11957 heartbeater.cc:503] Master 127.10.219.190:41825 was elected leader, sending a full tablet report...
I20240501 20:08:09.646387 11118 test_util.cc:268] Using random seed: 195410198
I20240501 20:08:09.661254 11681 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55118:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:08:09.662150 11681 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:08:09.686911 11773 tablet_service.cc:1467] Processing CreateTablet for tablet e3c5b2965cd04377b0d24e9fededf0f1 (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 20:08:09.687685 11773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3c5b2965cd04377b0d24e9fededf0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.689047 11921 tablet_service.cc:1467] Processing CreateTablet for tablet 931bdc7b4a9e42d6a7a36d050424057e (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 20:08:09.689342 11922 tablet_service.cc:1467] Processing CreateTablet for tablet e3c5b2965cd04377b0d24e9fededf0f1 (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 20:08:09.689680 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931bdc7b4a9e42d6a7a36d050424057e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.690129 11920 tablet_service.cc:1467] Processing CreateTablet for tablet e50657592d36432bac74f2c6a032ceae (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 20:08:09.690688 11920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50657592d36432bac74f2c6a032ceae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.691824 11769 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4553e4778b4dbf85b2bbb5f855247a (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 20:08:09.692586 11769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4553e4778b4dbf85b2bbb5f855247a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.695894 11922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3c5b2965cd04377b0d24e9fededf0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.696588 11917 tablet_service.cc:1467] Processing CreateTablet for tablet ff74f06417f44eb59a118c69770ab6fa (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 20:08:09.697177 11917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff74f06417f44eb59a118c69770ab6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.699528 11919 tablet_service.cc:1467] Processing CreateTablet for tablet ce9aa1175371401d8ca1db92b34cf504 (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 20:08:09.699533 11918 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4553e4778b4dbf85b2bbb5f855247a (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 20:08:09.700433 11771 tablet_service.cc:1467] Processing CreateTablet for tablet e50657592d36432bac74f2c6a032ceae (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 20:08:09.702589 11772 tablet_service.cc:1467] Processing CreateTablet for tablet 931bdc7b4a9e42d6a7a36d050424057e (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 20:08:09.703413 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9aa1175371401d8ca1db92b34cf504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.709271 11770 tablet_service.cc:1467] Processing CreateTablet for tablet ce9aa1175371401d8ca1db92b34cf504 (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 20:08:09.710551 11771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50657592d36432bac74f2c6a032ceae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.709127 11766 tablet_service.cc:1467] Processing CreateTablet for tablet 4f6abd6198bb4e39b00e9e5c7dfcaeb6 (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 20:08:09.711572 11918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4553e4778b4dbf85b2bbb5f855247a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.712579 11916 tablet_service.cc:1467] Processing CreateTablet for tablet 493dbd49298242408e327fd183b13034 (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 20:08:09.713285 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493dbd49298242408e327fd183b13034. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.715968 11767 tablet_service.cc:1467] Processing CreateTablet for tablet 493dbd49298242408e327fd183b13034 (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 20:08:09.711809 11915 tablet_service.cc:1467] Processing CreateTablet for tablet 4f6abd6198bb4e39b00e9e5c7dfcaeb6 (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 20:08:09.716650 11768 tablet_service.cc:1467] Processing CreateTablet for tablet ff74f06417f44eb59a118c69770ab6fa (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 20:08:09.722435 11772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931bdc7b4a9e42d6a7a36d050424057e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.725142 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f6abd6198bb4e39b00e9e5c7dfcaeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.729532 11770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9aa1175371401d8ca1db92b34cf504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.730495 11767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493dbd49298242408e327fd183b13034. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.733188 11768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff74f06417f44eb59a118c69770ab6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.733783 11766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f6abd6198bb4e39b00e9e5c7dfcaeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.746536 11977 tablet_bootstrap.cc:492] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc: Bootstrap starting.
I20240501 20:08:09.747651 11978 tablet_bootstrap.cc:492] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c: Bootstrap starting.
I20240501 20:08:09.758080 11978 tablet_bootstrap.cc:654] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.766660 11848 tablet_service.cc:1467] Processing CreateTablet for tablet e3c5b2965cd04377b0d24e9fededf0f1 (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 20:08:09.768661 11847 tablet_service.cc:1467] Processing CreateTablet for tablet 931bdc7b4a9e42d6a7a36d050424057e (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 20:08:09.768829 11846 tablet_service.cc:1467] Processing CreateTablet for tablet e50657592d36432bac74f2c6a032ceae (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 20:08:09.769109 11844 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4553e4778b4dbf85b2bbb5f855247a (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 20:08:09.770282 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3c5b2965cd04377b0d24e9fededf0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.770135 11842 tablet_service.cc:1467] Processing CreateTablet for tablet 493dbd49298242408e327fd183b13034 (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 20:08:09.770875 11843 tablet_service.cc:1467] Processing CreateTablet for tablet ff74f06417f44eb59a118c69770ab6fa (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 20:08:09.772212 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931bdc7b4a9e42d6a7a36d050424057e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.772490 11841 tablet_service.cc:1467] Processing CreateTablet for tablet 4f6abd6198bb4e39b00e9e5c7dfcaeb6 (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 20:08:09.772742 11843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff74f06417f44eb59a118c69770ab6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.771504 11845 tablet_service.cc:1467] Processing CreateTablet for tablet ce9aa1175371401d8ca1db92b34cf504 (DEFAULT_TABLE table=test-workload [id=5713b92dd48341ca868c05f4dcaf6c3e]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 20:08:09.774667 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50657592d36432bac74f2c6a032ceae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.775583 11842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493dbd49298242408e327fd183b13034. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.776907 11977 tablet_bootstrap.cc:654] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.777370 11844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4553e4778b4dbf85b2bbb5f855247a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.777921 11845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9aa1175371401d8ca1db92b34cf504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.778882 11841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f6abd6198bb4e39b00e9e5c7dfcaeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:09.790227 11978 tablet_bootstrap.cc:492] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c: No bootstrap required, opened a new log
I20240501 20:08:09.790529 11978 ts_tablet_manager.cc:1367] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.008s
I20240501 20:08:09.795717 11977 tablet_bootstrap.cc:492] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc: No bootstrap required, opened a new log
I20240501 20:08:09.795917 11977 ts_tablet_manager.cc:1367] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.004s
I20240501 20:08:09.796684 11977 raft_consensus.cc:348] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.796885 11977 raft_consensus.cc:374] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.797000 11977 raft_consensus.cc:729] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.797286 11977 consensus_queue.cc:260] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.809371 11981 tablet_bootstrap.cc:492] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348: Bootstrap starting.
I20240501 20:08:09.809360 11978 raft_consensus.cc:348] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.809706 11978 raft_consensus.cc:374] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.810509 11978 raft_consensus.cc:729] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dff34340a948dabf879205d1ea305c, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.810853 11978 consensus_queue.cc:260] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.813138 11981 tablet_bootstrap.cc:654] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.813905 11977 ts_tablet_manager.cc:1398] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc: Time spent starting tablet: real 0.018s user 0.000s sys 0.016s
I20240501 20:08:09.814427 11977 tablet_bootstrap.cc:492] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc: Bootstrap starting.
I20240501 20:08:09.817241 11977 tablet_bootstrap.cc:654] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.819813 11978 ts_tablet_manager.cc:1398] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c: Time spent starting tablet: real 0.029s user 0.007s sys 0.002s
I20240501 20:08:09.820219 11978 tablet_bootstrap.cc:492] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c: Bootstrap starting.
I20240501 20:08:09.822978 11978 tablet_bootstrap.cc:654] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.823338 11982 raft_consensus.cc:482] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:09.823668 11982 raft_consensus.cc:504] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.827124 11977 tablet_bootstrap.cc:492] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc: No bootstrap required, opened a new log
I20240501 20:08:09.827438 11977 ts_tablet_manager.cc:1367] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240501 20:08:09.828648 11977 raft_consensus.cc:348] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.829047 11977 raft_consensus.cc:374] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.829164 11977 raft_consensus.cc:729] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.829618 11977 consensus_queue.cc:260] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.832895 11977 ts_tablet_manager.cc:1398] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240501 20:08:09.834857 11978 tablet_bootstrap.cc:492] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c: No bootstrap required, opened a new log
I20240501 20:08:09.835129 11978 ts_tablet_manager.cc:1367] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20240501 20:08:09.836258 11978 raft_consensus.cc:348] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.836575 11978 raft_consensus.cc:374] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.836681 11978 raft_consensus.cc:729] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dff34340a948dabf879205d1ea305c, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.836980 11978 consensus_queue.cc:260] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.837575 11978 ts_tablet_manager.cc:1398] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:08:09.837946 11978 tablet_bootstrap.cc:492] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c: Bootstrap starting.
I20240501 20:08:09.841535 11977 tablet_bootstrap.cc:492] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc: Bootstrap starting.
I20240501 20:08:09.841653 11978 tablet_bootstrap.cc:654] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.845145 11977 tablet_bootstrap.cc:654] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.846884 11978 tablet_bootstrap.cc:492] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c: No bootstrap required, opened a new log
I20240501 20:08:09.850628 11978 ts_tablet_manager.cc:1367] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240501 20:08:09.851758 11978 raft_consensus.cc:348] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.852067 11978 raft_consensus.cc:374] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.852190 11978 raft_consensus.cc:729] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dff34340a948dabf879205d1ea305c, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.852584 11978 consensus_queue.cc:260] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.852608 11982 leader_election.cc:288] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381)
I20240501 20:08:09.859041 11978 ts_tablet_manager.cc:1398] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240501 20:08:09.859930 11978 tablet_bootstrap.cc:492] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c: Bootstrap starting.
I20240501 20:08:09.863373 11978 tablet_bootstrap.cc:654] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.873351 11978 tablet_bootstrap.cc:492] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c: No bootstrap required, opened a new log
I20240501 20:08:09.873916 11978 ts_tablet_manager.cc:1367] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240501 20:08:09.875463 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931bdc7b4a9e42d6a7a36d050424057e" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348" is_pre_election: true
I20240501 20:08:09.876405 11978 raft_consensus.cc:348] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.876802 11978 raft_consensus.cc:374] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.876921 11978 raft_consensus.cc:729] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dff34340a948dabf879205d1ea305c, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.877370 11978 consensus_queue.cc:260] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
W20240501 20:08:09.879319 11897 leader_election.cc:341] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:09.880697 11981 tablet_bootstrap.cc:492] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348: No bootstrap required, opened a new log
I20240501 20:08:09.880918 11981 ts_tablet_manager.cc:1367] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348: Time spent bootstrapping tablet: real 0.074s user 0.020s sys 0.046s
I20240501 20:08:09.881322 11978 ts_tablet_manager.cc:1398] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:08:09.881740 11978 tablet_bootstrap.cc:492] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c: Bootstrap starting.
I20240501 20:08:09.883435 11977 tablet_bootstrap.cc:492] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc: No bootstrap required, opened a new log
I20240501 20:08:09.883708 11977 ts_tablet_manager.cc:1367] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.023s
I20240501 20:08:09.883961 11978 tablet_bootstrap.cc:654] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.884232 11981 raft_consensus.cc:348] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.884507 11981 raft_consensus.cc:374] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.884843 11977 raft_consensus.cc:348] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.885167 11977 raft_consensus.cc:374] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.885334 11977 raft_consensus.cc:729] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.885727 11977 consensus_queue.cc:260] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.886494 11977 ts_tablet_manager.cc:1398] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:08:09.886878 11977 tablet_bootstrap.cc:492] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc: Bootstrap starting.
I20240501 20:08:09.885195 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931bdc7b4a9e42d6a7a36d050424057e" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c" is_pre_election: true
I20240501 20:08:09.884639 11981 raft_consensus.cc:729] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2676a1687a4171bb53684816779348, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.888037 11981 consensus_queue.cc:260] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
W20240501 20:08:09.888373 11895 leader_election.cc:341] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:09.888600 11895 leader_election.cc:302] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceb448c0562d444e8cbf5f256b20e9bc; no voters: 45dff34340a948dabf879205d1ea305c, 9a2676a1687a4171bb53684816779348
I20240501 20:08:09.889267 11982 raft_consensus.cc:2726] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:08:09.889724 11977 tablet_bootstrap.cc:654] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.892504 11981 ts_tablet_manager.cc:1398] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348: Time spent starting tablet: real 0.011s user 0.001s sys 0.005s
I20240501 20:08:09.894191 11981 tablet_bootstrap.cc:492] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348: Bootstrap starting.
I20240501 20:08:09.897167 11981 tablet_bootstrap.cc:654] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.901365 11978 tablet_bootstrap.cc:492] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c: No bootstrap required, opened a new log
I20240501 20:08:09.901624 11978 ts_tablet_manager.cc:1367] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.008s
I20240501 20:08:09.902420 11977 tablet_bootstrap.cc:492] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc: No bootstrap required, opened a new log
I20240501 20:08:09.902632 11977 ts_tablet_manager.cc:1367] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20240501 20:08:09.903012 11981 tablet_bootstrap.cc:492] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348: No bootstrap required, opened a new log
I20240501 20:08:09.902859 11978 raft_consensus.cc:348] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.903120 11978 raft_consensus.cc:374] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.903215 11981 ts_tablet_manager.cc:1367] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 20:08:09.903249 11978 raft_consensus.cc:729] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dff34340a948dabf879205d1ea305c, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.903579 11977 raft_consensus.cc:348] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.903545 11978 consensus_queue.cc:260] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.903785 11977 raft_consensus.cc:374] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.903882 11977 raft_consensus.cc:729] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.904021 11981 raft_consensus.cc:348] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.904284 11981 raft_consensus.cc:374] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.904201 11977 consensus_queue.cc:260] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.904340 11978 ts_tablet_manager.cc:1398] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:08:09.904383 11981 raft_consensus.cc:729] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2676a1687a4171bb53684816779348, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.904692 11978 tablet_bootstrap.cc:492] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c: Bootstrap starting.
I20240501 20:08:09.904754 11977 ts_tablet_manager.cc:1398] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:08:09.904697 11981 consensus_queue.cc:260] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.905056 11977 tablet_bootstrap.cc:492] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc: Bootstrap starting.
I20240501 20:08:09.905323 11981 ts_tablet_manager.cc:1398] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:09.905766 11981 tablet_bootstrap.cc:492] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348: Bootstrap starting.
I20240501 20:08:09.907305 11977 tablet_bootstrap.cc:654] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.907734 11981 tablet_bootstrap.cc:654] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.908402 11978 tablet_bootstrap.cc:654] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.913311 11981 tablet_bootstrap.cc:492] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348: No bootstrap required, opened a new log
I20240501 20:08:09.913547 11981 ts_tablet_manager.cc:1367] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240501 20:08:09.914525 11981 raft_consensus.cc:348] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.914798 11981 raft_consensus.cc:374] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.914902 11981 raft_consensus.cc:729] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2676a1687a4171bb53684816779348, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.915215 11981 consensus_queue.cc:260] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.915949 11981 ts_tablet_manager.cc:1398] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:09.916283 11981 tablet_bootstrap.cc:492] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348: Bootstrap starting.
I20240501 20:08:09.918419 11981 tablet_bootstrap.cc:654] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.921110 11977 tablet_bootstrap.cc:492] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc: No bootstrap required, opened a new log
I20240501 20:08:09.921248 11978 tablet_bootstrap.cc:492] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c: No bootstrap required, opened a new log
I20240501 20:08:09.921329 11977 ts_tablet_manager.cc:1367] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20240501 20:08:09.921416 11978 ts_tablet_manager.cc:1367] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.011s
I20240501 20:08:09.921501 11981 tablet_bootstrap.cc:492] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348: No bootstrap required, opened a new log
I20240501 20:08:09.921697 11981 ts_tablet_manager.cc:1367] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 20:08:09.922384 11981 raft_consensus.cc:348] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.922382 11978 raft_consensus.cc:348] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.922382 11977 raft_consensus.cc:348] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.922706 11981 raft_consensus.cc:374] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.922768 11978 raft_consensus.cc:374] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.922823 11977 raft_consensus.cc:374] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.922827 11981 raft_consensus.cc:729] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2676a1687a4171bb53684816779348, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.922883 11978 raft_consensus.cc:729] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dff34340a948dabf879205d1ea305c, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.922951 11977 raft_consensus.cc:729] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.923218 11977 consensus_queue.cc:260] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.923265 11978 consensus_queue.cc:260] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.923269 11981 consensus_queue.cc:260] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.923892 11978 ts_tablet_manager.cc:1398] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 20:08:09.923899 11977 ts_tablet_manager.cc:1398] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:08:09.924336 11978 tablet_bootstrap.cc:492] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c: Bootstrap starting.
I20240501 20:08:09.924343 11977 tablet_bootstrap.cc:492] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc: Bootstrap starting.
I20240501 20:08:09.924711 11981 ts_tablet_manager.cc:1398] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 20:08:09.925055 11981 tablet_bootstrap.cc:492] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348: Bootstrap starting.
I20240501 20:08:09.926576 11977 tablet_bootstrap.cc:654] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.926576 11978 tablet_bootstrap.cc:654] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.927090 11981 tablet_bootstrap.cc:654] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.929047 11982 raft_consensus.cc:482] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:09.929275 11982 raft_consensus.cc:504] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.930294 11982 leader_election.cc:288] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381)
I20240501 20:08:09.931229 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50657592d36432bac74f2c6a032ceae" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348" is_pre_election: true
I20240501 20:08:09.937304 11858 raft_consensus.cc:2454] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 0.
I20240501 20:08:09.931648 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50657592d36432bac74f2c6a032ceae" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c" is_pre_election: true
I20240501 20:08:09.937937 11978 tablet_bootstrap.cc:492] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c: No bootstrap required, opened a new log
I20240501 20:08:09.938143 11978 ts_tablet_manager.cc:1367] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20240501 20:08:09.938437 11897 leader_election.cc:302] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a2676a1687a4171bb53684816779348, ceb448c0562d444e8cbf5f256b20e9bc; no voters:
I20240501 20:08:09.938663 11977 tablet_bootstrap.cc:492] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc: No bootstrap required, opened a new log
I20240501 20:08:09.938840 11977 ts_tablet_manager.cc:1367] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240501 20:08:09.937901 11783 raft_consensus.cc:2454] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 0.
I20240501 20:08:09.939201 11978 raft_consensus.cc:348] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.939419 11978 raft_consensus.cc:374] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.939528 11978 raft_consensus.cc:729] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dff34340a948dabf879205d1ea305c, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.939615 11977 raft_consensus.cc:348] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.939826 11977 raft_consensus.cc:374] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.939823 11978 consensus_queue.cc:260] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.939994 11977 raft_consensus.cc:729] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.940352 11977 consensus_queue.cc:260] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.940510 11978 ts_tablet_manager.cc:1398] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:08:09.940738 11988 raft_consensus.cc:2781] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:09.940847 11988 raft_consensus.cc:482] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:09.940938 11978 tablet_bootstrap.cc:492] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c: Bootstrap starting.
I20240501 20:08:09.940994 11988 raft_consensus.cc:3037] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:09.943193 11978 tablet_bootstrap.cc:654] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.944792 11988 raft_consensus.cc:504] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.946625 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50657592d36432bac74f2c6a032ceae" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c"
I20240501 20:08:09.947041 11783 raft_consensus.cc:3037] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:09.948454 11983 raft_consensus.cc:482] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:09.948710 11983 raft_consensus.cc:504] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.950345 11983 leader_election.cc:288] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381), ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485)
I20240501 20:08:09.952819 11983 raft_consensus.cc:482] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:09.954988 11783 raft_consensus.cc:2454] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 1.
I20240501 20:08:09.955514 11983 raft_consensus.cc:504] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.956036 11977 ts_tablet_manager.cc:1398] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 20:08:09.956121 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50657592d36432bac74f2c6a032ceae" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348"
I20240501 20:08:09.963670 11988 leader_election.cc:288] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 election: Requested vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381)
I20240501 20:08:09.964392 11858 raft_consensus.cc:3037] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:09.965996 11983 leader_election.cc:288] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381), ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485)
I20240501 20:08:09.966540 11988 raft_consensus.cc:482] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:09.971406 11858 raft_consensus.cc:2454] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 1.
I20240501 20:08:09.971961 11977 tablet_bootstrap.cc:492] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc: Bootstrap starting.
I20240501 20:08:09.973610 11897 leader_election.cc:302] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a2676a1687a4171bb53684816779348, ceb448c0562d444e8cbf5f256b20e9bc; no voters:
I20240501 20:08:09.974483 11988 raft_consensus.cc:504] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.976482 11977 tablet_bootstrap.cc:654] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:09.976716 11988 leader_election.cc:288] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381)
I20240501 20:08:09.977252 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3c5b2965cd04377b0d24e9fededf0f1" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348" is_pre_election: true
W20240501 20:08:09.978166 11897 leader_election.cc:341] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:09.979310 11982 raft_consensus.cc:2781] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:09.980227 11982 raft_consensus.cc:686] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Becoming Leader. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Running, Role: LEADER
I20240501 20:08:09.982239 11982 consensus_queue.cc:237] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.988574 11978 tablet_bootstrap.cc:492] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c: No bootstrap required, opened a new log
I20240501 20:08:09.992098 11978 ts_tablet_manager.cc:1367] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.010s
I20240501 20:08:09.994011 11978 raft_consensus.cc:348] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.994786 12005 raft_consensus.cc:482] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:09.995249 11981 tablet_bootstrap.cc:492] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348: No bootstrap required, opened a new log
I20240501 20:08:09.995524 11981 ts_tablet_manager.cc:1367] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348: Time spent bootstrapping tablet: real 0.071s user 0.019s sys 0.038s
I20240501 20:08:09.996816 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3c5b2965cd04377b0d24e9fededf0f1" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c" is_pre_election: true
I20240501 20:08:09.995092 12005 raft_consensus.cc:504] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:09.997653 11978 raft_consensus.cc:374] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:09.997998 11978 raft_consensus.cc:729] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dff34340a948dabf879205d1ea305c, State: Initialized, Role: FOLLOWER
I20240501 20:08:09.998060 11783 raft_consensus.cc:2454] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 0.
I20240501 20:08:09.999270 11977 tablet_bootstrap.cc:492] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc: No bootstrap required, opened a new log
I20240501 20:08:09.999316 11895 leader_election.cc:302] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, ceb448c0562d444e8cbf5f256b20e9bc; no voters: 9a2676a1687a4171bb53684816779348
I20240501 20:08:09.999516 11977 ts_tablet_manager.cc:1367] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.013s
I20240501 20:08:10.000972 11988 raft_consensus.cc:2781] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:10.001549 11988 raft_consensus.cc:482] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:10.001287 11978 consensus_queue.cc:260] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.002112 11978 ts_tablet_manager.cc:1398] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 20:08:10.002975 12005 leader_election.cc:288] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381), ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485)
I20240501 20:08:10.003016 11988 raft_consensus.cc:3037] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.003878 11981 raft_consensus.cc:348] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.004000 11977 raft_consensus.cc:348] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.004684 11977 raft_consensus.cc:374] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:10.005475 11981 raft_consensus.cc:374] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:10.006414 11977 raft_consensus.cc:729] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Initialized, Role: FOLLOWER
I20240501 20:08:10.006747 11977 consensus_queue.cc:260] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.007666 11981 raft_consensus.cc:729] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2676a1687a4171bb53684816779348, State: Initialized, Role: FOLLOWER
I20240501 20:08:10.006975 11680 catalog_manager.cc:5484] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131). New cstate: current_term: 1 leader_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:10.008883 11977 ts_tablet_manager.cc:1398] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240501 20:08:10.008723 11981 consensus_queue.cc:260] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.011245 11981 ts_tablet_manager.cc:1398] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348: Time spent starting tablet: real 0.015s user 0.001s sys 0.002s
I20240501 20:08:10.011687 11988 raft_consensus.cc:504] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.013504 11977 tablet_bootstrap.cc:492] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc: Bootstrap starting.
I20240501 20:08:10.013782 11981 tablet_bootstrap.cc:492] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348: Bootstrap starting.
I20240501 20:08:10.014010 11988 leader_election.cc:288] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 election: Requested vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381)
I20240501 20:08:10.014645 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3c5b2965cd04377b0d24e9fededf0f1" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c"
I20240501 20:08:10.014655 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3c5b2965cd04377b0d24e9fededf0f1" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348"
I20240501 20:08:10.014891 11783 raft_consensus.cc:3037] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.015379 11858 raft_consensus.cc:3037] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.021560 11783 raft_consensus.cc:2454] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 1.
I20240501 20:08:10.022135 11895 leader_election.cc:302] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, ceb448c0562d444e8cbf5f256b20e9bc; no voters:
I20240501 20:08:10.022745 11988 raft_consensus.cc:2781] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:10.023648 11988 raft_consensus.cc:686] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Becoming Leader. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Running, Role: LEADER
I20240501 20:08:10.024088 11988 consensus_queue.cc:237] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.024602 11977 tablet_bootstrap.cc:654] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:10.024727 11981 tablet_bootstrap.cc:654] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:10.025853 11857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4553e4778b4dbf85b2bbb5f855247a" candidate_uuid: "45dff34340a948dabf879205d1ea305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348" is_pre_election: true
I20240501 20:08:10.026319 11857 raft_consensus.cc:2454] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45dff34340a948dabf879205d1ea305c in term 0.
I20240501 20:08:10.026831 11856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3c5b2965cd04377b0d24e9fededf0f1" candidate_uuid: "45dff34340a948dabf879205d1ea305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348" is_pre_election: true
I20240501 20:08:10.028159 11855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493dbd49298242408e327fd183b13034" candidate_uuid: "45dff34340a948dabf879205d1ea305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348" is_pre_election: true
I20240501 20:08:10.030406 11932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4553e4778b4dbf85b2bbb5f855247a" candidate_uuid: "45dff34340a948dabf879205d1ea305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" is_pre_election: true
I20240501 20:08:10.031440 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3c5b2965cd04377b0d24e9fededf0f1" candidate_uuid: "45dff34340a948dabf879205d1ea305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" is_pre_election: true
I20240501 20:08:10.032308 11930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493dbd49298242408e327fd183b13034" candidate_uuid: "45dff34340a948dabf879205d1ea305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" is_pre_election: true
I20240501 20:08:10.032719 11930 raft_consensus.cc:2454] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45dff34340a948dabf879205d1ea305c in term 0.
I20240501 20:08:10.033843 11748 leader_election.cc:302] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, 9a2676a1687a4171bb53684816779348; no voters:
W20240501 20:08:10.034539 11748 leader_election.cc:341] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:10.034884 12005 raft_consensus.cc:2781] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:10.035041 12005 raft_consensus.cc:482] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:10.035169 12005 raft_consensus.cc:3037] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.036151 11977 tablet_bootstrap.cc:492] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc: No bootstrap required, opened a new log
I20240501 20:08:10.036417 11977 ts_tablet_manager.cc:1367] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.011s
I20240501 20:08:10.021620 11858 raft_consensus.cc:2454] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 1.
W20240501 20:08:10.037240 11747 leader_election.cc:341] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:10.038822 11977 raft_consensus.cc:348] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.039182 11977 raft_consensus.cc:374] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:10.039335 11977 raft_consensus.cc:729] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Initialized, Role: FOLLOWER
I20240501 20:08:10.039180 11747 leader_election.cc:302] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45dff34340a948dabf879205d1ea305c; no voters: 9a2676a1687a4171bb53684816779348, ceb448c0562d444e8cbf5f256b20e9bc
I20240501 20:08:10.039669 11977 consensus_queue.cc:260] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.039952 11747 leader_election.cc:302] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, ceb448c0562d444e8cbf5f256b20e9bc; no voters: 9a2676a1687a4171bb53684816779348
I20240501 20:08:10.040072 11681 catalog_manager.cc:5484] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131). New cstate: current_term: 1 leader_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:10.040408 11977 ts_tablet_manager.cc:1398] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20240501 20:08:10.040437 11983 raft_consensus.cc:2726] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:08:10.040740 11983 raft_consensus.cc:2781] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:10.040880 11983 raft_consensus.cc:482] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:10.040776 12005 raft_consensus.cc:504] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.040997 11983 raft_consensus.cc:3037] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.042919 12005 leader_election.cc:288] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 election: Requested vote from peers 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381), ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485)
I20240501 20:08:10.043005 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4553e4778b4dbf85b2bbb5f855247a" candidate_uuid: "45dff34340a948dabf879205d1ea305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348"
I20240501 20:08:10.043494 11983 raft_consensus.cc:504] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.043716 11858 raft_consensus.cc:3037] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.044227 11983 leader_election.cc:288] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 election: Requested vote from peers 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381), ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485)
I20240501 20:08:10.045053 11855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493dbd49298242408e327fd183b13034" candidate_uuid: "45dff34340a948dabf879205d1ea305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348"
W20240501 20:08:10.045828 11748 leader_election.cc:341] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:10.046895 11981 tablet_bootstrap.cc:492] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348: No bootstrap required, opened a new log
I20240501 20:08:10.047091 11981 ts_tablet_manager.cc:1367] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.005s
I20240501 20:08:10.047899 11981 raft_consensus.cc:348] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.048132 11981 raft_consensus.cc:374] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:10.048229 11981 raft_consensus.cc:729] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2676a1687a4171bb53684816779348, State: Initialized, Role: FOLLOWER
I20240501 20:08:10.048521 11981 consensus_queue.cc:260] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.048986 11981 ts_tablet_manager.cc:1398] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:08:10.049373 11981 tablet_bootstrap.cc:492] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348: Bootstrap starting.
I20240501 20:08:10.049715 11858 raft_consensus.cc:2454] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45dff34340a948dabf879205d1ea305c in term 1.
I20240501 20:08:10.050876 11748 leader_election.cc:302] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, 9a2676a1687a4171bb53684816779348; no voters:
I20240501 20:08:10.051275 12013 raft_consensus.cc:2781] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:10.051911 12013 raft_consensus.cc:686] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 1 LEADER]: Becoming Leader. State: Replica: 45dff34340a948dabf879205d1ea305c, State: Running, Role: LEADER
I20240501 20:08:10.052275 11930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4553e4778b4dbf85b2bbb5f855247a" candidate_uuid: "45dff34340a948dabf879205d1ea305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc"
I20240501 20:08:10.052384 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493dbd49298242408e327fd183b13034" candidate_uuid: "45dff34340a948dabf879205d1ea305c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc"
I20240501 20:08:10.052596 11930 raft_consensus.cc:3037] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.052618 11931 raft_consensus.cc:3037] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.052434 12013 consensus_queue.cc:237] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.056370 11930 raft_consensus.cc:2454] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45dff34340a948dabf879205d1ea305c in term 1.
I20240501 20:08:10.057163 11931 raft_consensus.cc:2454] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45dff34340a948dabf879205d1ea305c in term 1.
I20240501 20:08:10.057819 11981 tablet_bootstrap.cc:654] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:10.057875 11747 leader_election.cc:302] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, ceb448c0562d444e8cbf5f256b20e9bc; no voters: 9a2676a1687a4171bb53684816779348
I20240501 20:08:10.058629 11983 raft_consensus.cc:2781] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:10.058858 11983 raft_consensus.cc:686] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 1 LEADER]: Becoming Leader. State: Replica: 45dff34340a948dabf879205d1ea305c, State: Running, Role: LEADER
I20240501 20:08:10.059194 11983 consensus_queue.cc:237] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.063846 11981 tablet_bootstrap.cc:492] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348: No bootstrap required, opened a new log
I20240501 20:08:10.064168 11981 ts_tablet_manager.cc:1367] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20240501 20:08:10.064189 11988 raft_consensus.cc:482] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:10.064409 11988 raft_consensus.cc:504] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.065224 11988 leader_election.cc:288] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381)
I20240501 20:08:10.065178 11981 raft_consensus.cc:348] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.065411 11981 raft_consensus.cc:374] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:10.065533 11981 raft_consensus.cc:729] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2676a1687a4171bb53684816779348, State: Initialized, Role: FOLLOWER
I20240501 20:08:10.065847 11981 consensus_queue.cc:260] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.066203 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9aa1175371401d8ca1db92b34cf504" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c" is_pre_election: true
I20240501 20:08:10.066481 11783 raft_consensus.cc:2454] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 0.
I20240501 20:08:10.066581 11981 ts_tablet_manager.cc:1398] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:10.066879 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9aa1175371401d8ca1db92b34cf504" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348" is_pre_election: true
I20240501 20:08:10.067112 11981 tablet_bootstrap.cc:492] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348: Bootstrap starting.
I20240501 20:08:10.067149 11895 leader_election.cc:302] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, ceb448c0562d444e8cbf5f256b20e9bc; no voters:
I20240501 20:08:10.067180 11858 raft_consensus.cc:2454] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 0.
I20240501 20:08:10.067593 11988 raft_consensus.cc:2781] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:10.067724 11988 raft_consensus.cc:482] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:10.067828 11988 raft_consensus.cc:3037] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.065871 11681 catalog_manager.cc:5484] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45dff34340a948dabf879205d1ea305c (127.10.219.129). New cstate: current_term: 1 leader_uuid: "45dff34340a948dabf879205d1ea305c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:10.069192 11981 tablet_bootstrap.cc:654] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:10.071108 11988 raft_consensus.cc:504] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.071807 11981 tablet_bootstrap.cc:492] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348: No bootstrap required, opened a new log
I20240501 20:08:10.072089 11981 ts_tablet_manager.cc:1367] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 20:08:10.072901 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9aa1175371401d8ca1db92b34cf504" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c"
I20240501 20:08:10.073171 11783 raft_consensus.cc:3037] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.073077 11981 raft_consensus.cc:348] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.073475 11981 raft_consensus.cc:374] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:10.073427 11988 leader_election.cc:288] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 election: Requested vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381)
I20240501 20:08:10.073592 11981 raft_consensus.cc:729] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2676a1687a4171bb53684816779348, State: Initialized, Role: FOLLOWER
I20240501 20:08:10.073429 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9aa1175371401d8ca1db92b34cf504" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348"
I20240501 20:08:10.073817 11858 raft_consensus.cc:3037] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.073930 11981 consensus_queue.cc:260] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.074575 11981 ts_tablet_manager.cc:1398] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:10.076885 11783 raft_consensus.cc:2454] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 1.
I20240501 20:08:10.076885 11858 raft_consensus.cc:2454] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 1.
I20240501 20:08:10.077428 11895 leader_election.cc:302] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, ceb448c0562d444e8cbf5f256b20e9bc; no voters:
I20240501 20:08:10.077998 11988 raft_consensus.cc:2781] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:10.078189 11988 raft_consensus.cc:686] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Becoming Leader. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Running, Role: LEADER
I20240501 20:08:10.078526 11988 consensus_queue.cc:237] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.080212 11681 catalog_manager.cc:5484] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45dff34340a948dabf879205d1ea305c (127.10.219.129). New cstate: current_term: 1 leader_uuid: "45dff34340a948dabf879205d1ea305c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:10.081545 11680 catalog_manager.cc:5484] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131). New cstate: current_term: 1 leader_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:10.131170 11994 raft_consensus.cc:482] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:10.131428 11994 raft_consensus.cc:504] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.132560 11994 leader_election.cc:288] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485)
I20240501 20:08:10.140513 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931bdc7b4a9e42d6a7a36d050424057e" candidate_uuid: "9a2676a1687a4171bb53684816779348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c" is_pre_election: true
I20240501 20:08:10.140867 11783 raft_consensus.cc:2454] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a2676a1687a4171bb53684816779348 in term 0.
I20240501 20:08:10.141378 11821 leader_election.cc:302] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, 9a2676a1687a4171bb53684816779348; no voters:
I20240501 20:08:10.141732 11994 raft_consensus.cc:2781] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:10.142024 11988 raft_consensus.cc:482] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:10.142040 11994 raft_consensus.cc:482] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:10.142434 11994 raft_consensus.cc:3037] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.142036 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931bdc7b4a9e42d6a7a36d050424057e" candidate_uuid: "9a2676a1687a4171bb53684816779348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" is_pre_election: true
I20240501 20:08:10.142391 11988 raft_consensus.cc:504] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.142710 11931 raft_consensus.cc:2454] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a2676a1687a4171bb53684816779348 in term 0.
I20240501 20:08:10.143486 11988 leader_election.cc:288] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381)
I20240501 20:08:10.143846 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff74f06417f44eb59a118c69770ab6fa" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348" is_pre_election: true
I20240501 20:08:10.144086 11858 raft_consensus.cc:2454] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 0.
I20240501 20:08:10.144521 11897 leader_election.cc:302] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a2676a1687a4171bb53684816779348, ceb448c0562d444e8cbf5f256b20e9bc; no voters:
I20240501 20:08:10.145094 11988 raft_consensus.cc:2781] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:10.145269 11988 raft_consensus.cc:482] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:10.145411 11988 raft_consensus.cc:3037] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.145617 11994 raft_consensus.cc:504] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.146896 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff74f06417f44eb59a118c69770ab6fa" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c" is_pre_election: true
I20240501 20:08:10.146976 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931bdc7b4a9e42d6a7a36d050424057e" candidate_uuid: "9a2676a1687a4171bb53684816779348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc"
I20240501 20:08:10.147230 11783 raft_consensus.cc:2454] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 0.
I20240501 20:08:10.147614 11994 leader_election.cc:288] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [CANDIDATE]: Term 1 election: Requested vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485)
I20240501 20:08:10.147526 11782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931bdc7b4a9e42d6a7a36d050424057e" candidate_uuid: "9a2676a1687a4171bb53684816779348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c"
I20240501 20:08:10.148005 11782 raft_consensus.cc:3037] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.149201 11988 raft_consensus.cc:504] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.150116 11988 leader_election.cc:288] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 election: Requested vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381)
I20240501 20:08:10.150770 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff74f06417f44eb59a118c69770ab6fa" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c"
I20240501 20:08:10.150911 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff74f06417f44eb59a118c69770ab6fa" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348"
I20240501 20:08:10.151160 11783 raft_consensus.cc:3037] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.147228 11931 raft_consensus.cc:3037] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.151175 11858 raft_consensus.cc:3037] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.151188 11782 raft_consensus.cc:2454] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2676a1687a4171bb53684816779348 in term 1.
I20240501 20:08:10.153404 11821 leader_election.cc:302] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, 9a2676a1687a4171bb53684816779348; no voters:
I20240501 20:08:10.153931 11994 raft_consensus.cc:2781] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:10.154755 11858 raft_consensus.cc:2454] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 1.
I20240501 20:08:10.154883 11783 raft_consensus.cc:2454] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 1.
I20240501 20:08:10.155059 11931 raft_consensus.cc:2454] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2676a1687a4171bb53684816779348 in term 1.
I20240501 20:08:10.155308 11897 leader_election.cc:302] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a2676a1687a4171bb53684816779348, ceb448c0562d444e8cbf5f256b20e9bc; no voters:
I20240501 20:08:10.155743 11988 raft_consensus.cc:2781] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:10.156002 11988 raft_consensus.cc:686] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Becoming Leader. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Running, Role: LEADER
I20240501 20:08:10.156482 11988 consensus_queue.cc:237] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.160439 11982 raft_consensus.cc:482] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:10.160809 11982 raft_consensus.cc:504] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.162070 11982 leader_election.cc:288] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381)
I20240501 20:08:10.163118 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6abd6198bb4e39b00e9e5c7dfcaeb6" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348" is_pre_election: true
I20240501 20:08:10.163111 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6abd6198bb4e39b00e9e5c7dfcaeb6" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c" is_pre_election: true
I20240501 20:08:10.163590 11858 raft_consensus.cc:2454] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 0.
I20240501 20:08:10.163846 11783 raft_consensus.cc:2454] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 0.
I20240501 20:08:10.164379 11897 leader_election.cc:302] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a2676a1687a4171bb53684816779348, ceb448c0562d444e8cbf5f256b20e9bc; no voters:
I20240501 20:08:10.165292 11982 raft_consensus.cc:2781] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:10.165524 11982 raft_consensus.cc:482] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:10.165664 11982 raft_consensus.cc:3037] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.167327 11994 raft_consensus.cc:686] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 1 LEADER]: Becoming Leader. State: Replica: 9a2676a1687a4171bb53684816779348, State: Running, Role: LEADER
I20240501 20:08:10.167804 11994 consensus_queue.cc:237] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.168267 11680 catalog_manager.cc:5484] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131). New cstate: current_term: 1 leader_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:10.169350 11982 raft_consensus.cc:504] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.170547 11982 leader_election.cc:288] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 election: Requested vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381)
I20240501 20:08:10.170704 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6abd6198bb4e39b00e9e5c7dfcaeb6" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2676a1687a4171bb53684816779348"
I20240501 20:08:10.170920 11858 raft_consensus.cc:3037] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.172540 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6abd6198bb4e39b00e9e5c7dfcaeb6" candidate_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c"
I20240501 20:08:10.172791 11783 raft_consensus.cc:3037] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.172799 11681 catalog_manager.cc:5484] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a2676a1687a4171bb53684816779348 (127.10.219.130). New cstate: current_term: 1 leader_uuid: "9a2676a1687a4171bb53684816779348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:10.174129 11858 raft_consensus.cc:2454] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 1.
I20240501 20:08:10.174890 11897 leader_election.cc:302] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a2676a1687a4171bb53684816779348, ceb448c0562d444e8cbf5f256b20e9bc; no voters:
I20240501 20:08:10.175240 11982 raft_consensus.cc:2781] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:10.175412 11982 raft_consensus.cc:686] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Becoming Leader. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Running, Role: LEADER
I20240501 20:08:10.175799 11982 consensus_queue.cc:237] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.176126 11783 raft_consensus.cc:2454] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb448c0562d444e8cbf5f256b20e9bc in term 1.
I20240501 20:08:10.179297 11681 catalog_manager.cc:5484] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131). New cstate: current_term: 1 leader_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:10.230337 11994 raft_consensus.cc:482] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:10.230609 11994 raft_consensus.cc:504] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.231321 11994 leader_election.cc:288] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485)
I20240501 20:08:10.231976 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493dbd49298242408e327fd183b13034" candidate_uuid: "9a2676a1687a4171bb53684816779348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dff34340a948dabf879205d1ea305c" is_pre_election: true
I20240501 20:08:10.231966 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493dbd49298242408e327fd183b13034" candidate_uuid: "9a2676a1687a4171bb53684816779348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" is_pre_election: true
I20240501 20:08:10.232388 11931 raft_consensus.cc:2379] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a2676a1687a4171bb53684816779348 in current term 1: Already voted for candidate 45dff34340a948dabf879205d1ea305c in this term.
I20240501 20:08:10.232811 11822 leader_election.cc:302] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a2676a1687a4171bb53684816779348; no voters: 45dff34340a948dabf879205d1ea305c, ceb448c0562d444e8cbf5f256b20e9bc
I20240501 20:08:10.233114 11994 raft_consensus.cc:3037] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:10.235229 11994 raft_consensus.cc:2726] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:08:10.269379 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:10.273224 12025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:10.273866 12026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:10.274557 12028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:10.275066 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:10.275354 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:10.275439 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:10.275511 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594090275484 us; error 0 us; skew 500 ppm
I20240501 20:08:10.275729 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:10.277226 11118 webserver.cc:460] Webserver started at http://127.10.219.132:34811/ using document root <none> and password file <none>
I20240501 20:08:10.277503 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:10.277595 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:10.277778 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:10.278604 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-3-root/instance:
uuid: "cf3f4b94559b42808d47ba2088659b0f"
format_stamp: "Formatted at 2024-05-01 20:08:10 on dist-test-slave-n6pz"
I20240501 20:08:10.281657 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:10.283735 12033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:10.284161 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 20:08:10.284286 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-3-root
uuid: "cf3f4b94559b42808d47ba2088659b0f"
format_stamp: "Formatted at 2024-05-01 20:08:10 on dist-test-slave-n6pz"
I20240501 20:08:10.284443 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714594084913313-11118-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:10.298792 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:10.299525 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:10.300213 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:10.301316 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:10.301407 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:10.301538 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:10.301637 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:10.323765 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.132:40395
I20240501 20:08:10.323801 12095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.132:40395 every 8 connection(s)
I20240501 20:08:10.332415 12096 heartbeater.cc:347] Connected to a master server at 127.10.219.190:41825
I20240501 20:08:10.332592 12096 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:10.333063 12096 heartbeater.cc:511] Master 127.10.219.190:41825 requested a full tablet report, sending...
I20240501 20:08:10.334014 11681 ts_manager.cc:194] Registered new tserver with Master: cf3f4b94559b42808d47ba2088659b0f (127.10.219.132:40395)
I20240501 20:08:10.334898 11681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55122
I20240501 20:08:10.374532 11734 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:10.386279 11931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3c5b2965cd04377b0d24e9fededf0f1"
dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc"
mode: GRACEFUL
new_leader_uuid: "9a2676a1687a4171bb53684816779348"
from {username='slave'} at 127.0.0.1:49994
I20240501 20:08:10.386679 11931 raft_consensus.cc:595] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Received request to transfer leadership to TS 9a2676a1687a4171bb53684816779348
I20240501 20:08:10.388525 11931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e50657592d36432bac74f2c6a032ceae"
dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc"
mode: GRACEFUL
new_leader_uuid: "9a2676a1687a4171bb53684816779348"
from {username='slave'} at 127.0.0.1:49994
I20240501 20:08:10.388797 11931 raft_consensus.cc:595] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Received request to transfer leadership to TS 9a2676a1687a4171bb53684816779348
I20240501 20:08:10.390064 11931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff74f06417f44eb59a118c69770ab6fa"
dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc"
mode: GRACEFUL
new_leader_uuid: "45dff34340a948dabf879205d1ea305c"
from {username='slave'} at 127.0.0.1:49994
I20240501 20:08:10.390307 11931 raft_consensus.cc:595] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Received request to transfer leadership to TS 45dff34340a948dabf879205d1ea305c
I20240501 20:08:10.391213 11734 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 20:08:10.391566 11734 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:10.456281 12013 consensus_queue.cc:1035] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.461174 12013 consensus_queue.cc:1035] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.473680 11982 consensus_queue.cc:1035] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.476823 11988 raft_consensus.cc:979] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc: : Instructing follower 9a2676a1687a4171bb53684816779348 to start an election
I20240501 20:08:10.477113 11982 raft_consensus.cc:1067] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Signalling peer 9a2676a1687a4171bb53684816779348 to start an election
I20240501 20:08:10.478369 11855 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e50657592d36432bac74f2c6a032ceae"
dest_uuid: "9a2676a1687a4171bb53684816779348"
from {username='slave'} at 127.0.0.1:42854
I20240501 20:08:10.478663 11855 raft_consensus.cc:482] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:10.478814 11855 raft_consensus.cc:3037] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:10.479280 11988 consensus_queue.cc:1035] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.482491 11855 raft_consensus.cc:504] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.483409 11855 leader_election.cc:288] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [CANDIDATE]: Term 2 election: Requested vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485)
I20240501 20:08:10.483765 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50657592d36432bac74f2c6a032ceae" candidate_uuid: "9a2676a1687a4171bb53684816779348" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "45dff34340a948dabf879205d1ea305c"
I20240501 20:08:10.483999 11783 raft_consensus.cc:3037] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:10.484094 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50657592d36432bac74f2c6a032ceae" candidate_uuid: "9a2676a1687a4171bb53684816779348" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc"
I20240501 20:08:10.484316 11931 raft_consensus.cc:3032] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:10.484441 11931 raft_consensus.cc:729] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Running, Role: LEADER
I20240501 20:08:10.484861 11931 consensus_queue.cc:260] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.485586 11931 raft_consensus.cc:3037] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:10.487016 11783 raft_consensus.cc:2454] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2676a1687a4171bb53684816779348 in term 2.
I20240501 20:08:10.488018 11821 leader_election.cc:302] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, 9a2676a1687a4171bb53684816779348; no voters:
I20240501 20:08:10.488170 11931 raft_consensus.cc:2454] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2676a1687a4171bb53684816779348 in term 2.
I20240501 20:08:10.488441 11994 raft_consensus.cc:2781] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:10.488616 11994 raft_consensus.cc:686] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 2 LEADER]: Becoming Leader. State: Replica: 9a2676a1687a4171bb53684816779348, State: Running, Role: LEADER
I20240501 20:08:10.488910 11994 consensus_queue.cc:237] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.491309 11681 catalog_manager.cc:5484] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 reported cstate change: term changed from 1 to 2, leader changed from ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131) to 9a2676a1687a4171bb53684816779348 (127.10.219.130). New cstate: current_term: 2 leader_uuid: "9a2676a1687a4171bb53684816779348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:10.510641 12107 consensus_queue.cc:1035] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.515106 12107 consensus_queue.cc:1035] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.519228 11988 raft_consensus.cc:979] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc: : Instructing follower 9a2676a1687a4171bb53684816779348 to start an election
I20240501 20:08:10.519443 11988 raft_consensus.cc:1067] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Signalling peer 9a2676a1687a4171bb53684816779348 to start an election
I20240501 20:08:10.520207 11855 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3c5b2965cd04377b0d24e9fededf0f1"
dest_uuid: "9a2676a1687a4171bb53684816779348"
from {username='slave'} at 127.0.0.1:42854
I20240501 20:08:10.520447 11855 raft_consensus.cc:482] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:10.520566 11855 raft_consensus.cc:3037] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:10.522823 11855 raft_consensus.cc:504] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.523730 11855 leader_election.cc:288] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [CANDIDATE]: Term 2 election: Requested vote from peers 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653), ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485)
I20240501 20:08:10.524248 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3c5b2965cd04377b0d24e9fededf0f1" candidate_uuid: "9a2676a1687a4171bb53684816779348" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "45dff34340a948dabf879205d1ea305c"
I20240501 20:08:10.524247 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3c5b2965cd04377b0d24e9fededf0f1" candidate_uuid: "9a2676a1687a4171bb53684816779348" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc"
I20240501 20:08:10.524567 11931 raft_consensus.cc:3032] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:10.524565 11783 raft_consensus.cc:3037] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:10.524699 11931 raft_consensus.cc:729] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Running, Role: LEADER
I20240501 20:08:10.525017 11931 consensus_queue.cc:260] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.525444 11931 raft_consensus.cc:3037] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:10.528045 11783 raft_consensus.cc:2454] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2676a1687a4171bb53684816779348 in term 2.
I20240501 20:08:10.528329 11931 raft_consensus.cc:2454] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a2676a1687a4171bb53684816779348 in term 2.
I20240501 20:08:10.528676 11821 leader_election.cc:302] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, 9a2676a1687a4171bb53684816779348; no voters:
I20240501 20:08:10.529012 12021 raft_consensus.cc:2781] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:10.529176 12021 raft_consensus.cc:686] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [term 2 LEADER]: Becoming Leader. State: Replica: 9a2676a1687a4171bb53684816779348, State: Running, Role: LEADER
I20240501 20:08:10.529495 12021 consensus_queue.cc:237] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.532030 11680 catalog_manager.cc:5484] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 reported cstate change: term changed from 1 to 2, leader changed from ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131) to 9a2676a1687a4171bb53684816779348 (127.10.219.130). New cstate: current_term: 2 leader_uuid: "9a2676a1687a4171bb53684816779348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:10.616612 11988 consensus_queue.cc:1035] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.622076 12112 consensus_queue.cc:1035] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.623895 12112 consensus_queue.cc:1035] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.628793 12107 consensus_queue.cc:1035] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.643644 12013 consensus_queue.cc:1035] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.644429 12107 consensus_queue.cc:1035] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.646550 12021 consensus_queue.cc:1035] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.649220 12107 raft_consensus.cc:979] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc: : Instructing follower 45dff34340a948dabf879205d1ea305c to start an election
I20240501 20:08:10.649415 12107 raft_consensus.cc:1067] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Signalling peer 45dff34340a948dabf879205d1ea305c to start an election
I20240501 20:08:10.652359 11781 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff74f06417f44eb59a118c69770ab6fa"
dest_uuid: "45dff34340a948dabf879205d1ea305c"
from {username='slave'} at 127.0.0.1:56702
I20240501 20:08:10.652676 11781 raft_consensus.cc:482] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:10.652838 11781 raft_consensus.cc:3037] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:10.653169 11982 consensus_queue.cc:1035] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.653757 11994 consensus_queue.cc:1035] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.654727 11983 consensus_queue.cc:1035] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:10.659991 11781 raft_consensus.cc:504] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.666133 11930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff74f06417f44eb59a118c69770ab6fa" candidate_uuid: "45dff34340a948dabf879205d1ea305c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ceb448c0562d444e8cbf5f256b20e9bc"
I20240501 20:08:10.666079 11855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff74f06417f44eb59a118c69770ab6fa" candidate_uuid: "45dff34340a948dabf879205d1ea305c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9a2676a1687a4171bb53684816779348"
I20240501 20:08:10.666807 11930 raft_consensus.cc:3032] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:10.666923 11930 raft_consensus.cc:729] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ceb448c0562d444e8cbf5f256b20e9bc, State: Running, Role: LEADER
I20240501 20:08:10.666525 11781 leader_election.cc:288] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 2 election: Requested vote from peers 9a2676a1687a4171bb53684816779348 (127.10.219.130:41381), ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131:36485)
I20240501 20:08:10.667950 11930 consensus_queue.cc:260] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.668500 11930 raft_consensus.cc:3037] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:10.673053 11930 raft_consensus.cc:2454] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45dff34340a948dabf879205d1ea305c in term 2.
I20240501 20:08:10.673528 11747 leader_election.cc:302] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45dff34340a948dabf879205d1ea305c, ceb448c0562d444e8cbf5f256b20e9bc; no voters: 9a2676a1687a4171bb53684816779348
I20240501 20:08:10.674108 12005 raft_consensus.cc:2781] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:10.674329 12005 raft_consensus.cc:686] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 2 LEADER]: Becoming Leader. State: Replica: 45dff34340a948dabf879205d1ea305c, State: Running, Role: LEADER
I20240501 20:08:10.674640 12005 consensus_queue.cc:237] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } }
I20240501 20:08:10.677199 11681 catalog_manager.cc:5484] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c reported cstate change: term changed from 1 to 2, leader changed from ceb448c0562d444e8cbf5f256b20e9bc (127.10.219.131) to 45dff34340a948dabf879205d1ea305c (127.10.219.129). New cstate: current_term: 2 leader_uuid: "45dff34340a948dabf879205d1ea305c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:10.910475 11782 raft_consensus.cc:1261] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c [term 2 FOLLOWER]: Refusing update from remote peer 9a2676a1687a4171bb53684816779348: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:08:10.911129 12119 consensus_queue.cc:1035] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:10.914253 11930 raft_consensus.cc:1261] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Refusing update from remote peer 9a2676a1687a4171bb53684816779348: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:08:10.914916 12119 consensus_queue.cc:1035] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:10.978417 11782 raft_consensus.cc:1261] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 2 FOLLOWER]: Refusing update from remote peer 9a2676a1687a4171bb53684816779348: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:08:10.979182 12119 consensus_queue.cc:1035] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:10.982497 11930 raft_consensus.cc:1261] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Refusing update from remote peer 9a2676a1687a4171bb53684816779348: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:08:10.983369 12119 consensus_queue.cc:1035] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:11.071772 11858 raft_consensus.cc:3037] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:11.074685 11858 raft_consensus.cc:1261] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [term 2 FOLLOWER]: Refusing update from remote peer 45dff34340a948dabf879205d1ea305c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:08:11.075537 12005 consensus_queue.cc:1035] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:11.079392 11930 raft_consensus.cc:1261] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Refusing update from remote peer 45dff34340a948dabf879205d1ea305c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:08:11.080236 12005 consensus_queue.cc:1035] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:11.336328 12096 heartbeater.cc:503] Master 127.10.219.190:41825 was elected leader, sending a full tablet report...
I20240501 20:08:12.392072 11734 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:12.393532 11734 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:12.393747 11734 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:14.394296 11734 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:14.395823 11734 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:14.396073 11734 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:15.392683 11931 consensus_queue.cc:237] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } }
I20240501 20:08:15.396510 11858 raft_consensus.cc:1261] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Refusing update from remote peer ceb448c0562d444e8cbf5f256b20e9bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:15.396915 11782 raft_consensus.cc:1261] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Refusing update from remote peer ceb448c0562d444e8cbf5f256b20e9bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:15.397621 12112 consensus_queue.cc:1035] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:15.398455 12128 consensus_queue.cc:1035] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:15.401691 12129 raft_consensus.cc:2932] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.402472 11782 raft_consensus.cc:2932] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.402470 11858 raft_consensus.cc:2932] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
W20240501 20:08:15.408210 11898 consensus_peers.cc:485] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc -> Peer cf3f4b94559b42808d47ba2088659b0f (127.10.219.132:40395): Couldn't send request to peer cf3f4b94559b42808d47ba2088659b0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f6abd6198bb4e39b00e9e5c7dfcaeb6. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:15.409308 11679 catalog_manager.cc:5484] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c reported cstate change: config changed from index -1 to 2, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New cstate: current_term: 1 leader_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.418970 11782 consensus_queue.cc:237] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } attrs { replace: true } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } }
I20240501 20:08:15.422050 11931 raft_consensus.cc:1261] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Refusing update from remote peer 45dff34340a948dabf879205d1ea305c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:15.422067 11858 raft_consensus.cc:1261] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Refusing update from remote peer 45dff34340a948dabf879205d1ea305c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:15.422772 12124 consensus_queue.cc:1035] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:15.423420 12124 consensus_queue.cc:1035] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:15.427305 12124 raft_consensus.cc:2932] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } attrs { replace: true } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.427626 11858 raft_consensus.cc:2932] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } attrs { replace: true } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.434911 11680 catalog_manager.cc:5484] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 reported cstate change: config changed from index -1 to 2, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New cstate: current_term: 1 leader_uuid: "45dff34340a948dabf879205d1ea305c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } attrs { replace: true } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.435086 11931 raft_consensus.cc:2932] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } attrs { replace: true } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
W20240501 20:08:15.438333 11749 consensus_peers.cc:485] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c -> Peer cf3f4b94559b42808d47ba2088659b0f (127.10.219.132:40395): Couldn't send request to peer cf3f4b94559b42808d47ba2088659b0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 493dbd49298242408e327fd183b13034. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:15.439636 11782 consensus_queue.cc:237] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } attrs { replace: true } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } }
I20240501 20:08:15.442247 11931 raft_consensus.cc:1261] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Refusing update from remote peer 45dff34340a948dabf879205d1ea305c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:08:15.442276 11858 raft_consensus.cc:1261] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [term 2 FOLLOWER]: Refusing update from remote peer 45dff34340a948dabf879205d1ea305c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240501 20:08:15.442476 11749 consensus_peers.cc:485] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c -> Peer cf3f4b94559b42808d47ba2088659b0f (127.10.219.132:40395): Couldn't send request to peer cf3f4b94559b42808d47ba2088659b0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff74f06417f44eb59a118c69770ab6fa. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:15.442834 12141 consensus_queue.cc:1035] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:08:15.443200 12140 consensus_queue.cc:1035] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:08:15.446815 12141 raft_consensus.cc:2932] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } attrs { replace: true } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.447621 11931 raft_consensus.cc:2932] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } attrs { replace: true } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.448796 11858 raft_consensus.cc:2932] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } attrs { replace: true } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.453951 11681 catalog_manager.cc:5484] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c reported cstate change: config changed from index -1 to 3, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New cstate: current_term: 2 leader_uuid: "45dff34340a948dabf879205d1ea305c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:15.455662 11930 consensus_queue.cc:237] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } attrs { replace: true } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } }
I20240501 20:08:15.458925 11782 raft_consensus.cc:1261] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Refusing update from remote peer ceb448c0562d444e8cbf5f256b20e9bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:08:15.459411 11898 consensus_peers.cc:485] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc -> Peer cf3f4b94559b42808d47ba2088659b0f (127.10.219.132:40395): Couldn't send request to peer cf3f4b94559b42808d47ba2088659b0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce9aa1175371401d8ca1db92b34cf504. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:15.459606 11858 raft_consensus.cc:1261] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Refusing update from remote peer ceb448c0562d444e8cbf5f256b20e9bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:15.459836 12112 consensus_queue.cc:1035] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:15.461266 12112 consensus_queue.cc:1035] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:15.462838 12128 raft_consensus.cc:2932] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } attrs { replace: true } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.463640 11782 raft_consensus.cc:2932] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } attrs { replace: true } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.465178 11858 raft_consensus.cc:2932] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } attrs { replace: true } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.469643 11680 catalog_manager.cc:5484] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c reported cstate change: config changed from index -1 to 2, NON_VOTER cf3f4b94559b42808d47ba2088659b0f (127.10.219.132) added. New cstate: current_term: 1 leader_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "45dff34340a948dabf879205d1ea305c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 36653 } } peers { permanent_uuid: "9a2676a1687a4171bb53684816779348" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 41381 } attrs { replace: true } } peers { permanent_uuid: "ceb448c0562d444e8cbf5f256b20e9bc" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36485 } } peers { permanent_uuid: "cf3f4b94559b42808d47ba2088659b0f" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 40395 } attrs { promote: true } } }
I20240501 20:08:15.585712 11118 tablet_server.cc:178] TabletServer@127.10.219.129:0 shutting down...
W20240501 20:08:15.594401 11665 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20240501 20:08:15.596951 11729 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 20:08:15.597476 11729 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 20:08:15.598888 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:15.599252 11118 tablet_replica.cc:330] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c: stopping tablet replica
I20240501 20:08:15.599575 11118 raft_consensus.cc:2229] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.599977 11118 raft_consensus.cc:2258] T ce9aa1175371401d8ca1db92b34cf504 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.601011 11118 tablet_replica.cc:330] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c: stopping tablet replica
I20240501 20:08:15.601219 11118 raft_consensus.cc:2229] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:15.601531 11118 raft_consensus.cc:2258] T 6a4553e4778b4dbf85b2bbb5f855247a P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 20:08:15.602099 11729 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 20:08:15.602186 11729 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 20:08:15.602190 11118 tablet_replica.cc:330] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c: stopping tablet replica
I20240501 20:08:15.602425 11118 raft_consensus.cc:2229] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.602622 11118 raft_consensus.cc:2258] T 931bdc7b4a9e42d6a7a36d050424057e P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.603209 11118 tablet_replica.cc:330] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c: stopping tablet replica
I20240501 20:08:15.603410 11118 raft_consensus.cc:2229] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.603621 11118 raft_consensus.cc:2258] T e3c5b2965cd04377b0d24e9fededf0f1 P 45dff34340a948dabf879205d1ea305c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.604107 11118 tablet_replica.cc:330] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c: stopping tablet replica
I20240501 20:08:15.604301 11118 raft_consensus.cc:2229] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.604552 11118 raft_consensus.cc:2258] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.605286 11118 tablet_replica.cc:330] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c: stopping tablet replica
I20240501 20:08:15.605481 11118 raft_consensus.cc:2229] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.605670 11118 raft_consensus.cc:2258] T e50657592d36432bac74f2c6a032ceae P 45dff34340a948dabf879205d1ea305c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.606230 11118 tablet_replica.cc:330] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c: stopping tablet replica
I20240501 20:08:15.606431 11118 raft_consensus.cc:2229] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:08:15.606952 11118 raft_consensus.cc:2258] T ff74f06417f44eb59a118c69770ab6fa P 45dff34340a948dabf879205d1ea305c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.607899 11118 tablet_replica.cc:330] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c: stopping tablet replica
I20240501 20:08:15.608132 11118 raft_consensus.cc:2229] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:15.608557 11118 raft_consensus.cc:2258] T 493dbd49298242408e327fd183b13034 P 45dff34340a948dabf879205d1ea305c [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 20:08:15.609717 11821 consensus_peers.cc:485] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 -> Peer 45dff34340a948dabf879205d1ea305c (127.10.219.129:36653): Couldn't send request to peer 45dff34340a948dabf879205d1ea305c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:15.629686 11118 tablet_server.cc:194] TabletServer@127.10.219.129:0 shutdown complete.
I20240501 20:08:15.635205 11118 tablet_server.cc:178] TabletServer@127.10.219.130:0 shutting down...
I20240501 20:08:15.646224 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:15.646754 11118 tablet_replica.cc:330] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348: stopping tablet replica
I20240501 20:08:15.646965 11118 raft_consensus.cc:2229] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.647217 11118 raft_consensus.cc:2258] T ce9aa1175371401d8ca1db92b34cf504 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.648227 11118 tablet_replica.cc:330] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348: stopping tablet replica
I20240501 20:08:15.648419 11118 raft_consensus.cc:2229] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.648671 11118 raft_consensus.cc:2258] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.649494 11118 tablet_replica.cc:330] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348: stopping tablet replica
I20240501 20:08:15.649703 11118 raft_consensus.cc:2229] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:08:15.650018 11118 raft_consensus.cc:2258] T e50657592d36432bac74f2c6a032ceae P 9a2676a1687a4171bb53684816779348 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.650664 11118 tablet_replica.cc:330] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348: stopping tablet replica
I20240501 20:08:15.650842 11118 raft_consensus.cc:2229] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:08:15.651187 11118 raft_consensus.cc:2258] T e3c5b2965cd04377b0d24e9fededf0f1 P 9a2676a1687a4171bb53684816779348 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.651748 11118 tablet_replica.cc:330] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348: stopping tablet replica
I20240501 20:08:15.651937 11118 raft_consensus.cc:2229] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:15.652240 11118 raft_consensus.cc:2258] T 931bdc7b4a9e42d6a7a36d050424057e P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.652813 11118 tablet_replica.cc:330] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348: stopping tablet replica
I20240501 20:08:15.652983 11118 raft_consensus.cc:2229] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.653167 11118 raft_consensus.cc:2258] T 6a4553e4778b4dbf85b2bbb5f855247a P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.653671 11118 tablet_replica.cc:330] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348: stopping tablet replica
I20240501 20:08:15.653864 11118 raft_consensus.cc:2229] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.654088 11118 raft_consensus.cc:2258] T 493dbd49298242408e327fd183b13034 P 9a2676a1687a4171bb53684816779348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.654865 11118 tablet_replica.cc:330] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348: stopping tablet replica
I20240501 20:08:15.655020 11118 raft_consensus.cc:2229] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.655231 11118 raft_consensus.cc:2258] T ff74f06417f44eb59a118c69770ab6fa P 9a2676a1687a4171bb53684816779348 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.662406 11118 tablet_server.cc:194] TabletServer@127.10.219.130:0 shutdown complete.
I20240501 20:08:15.667691 11118 tablet_server.cc:178] TabletServer@127.10.219.131:0 shutting down...
W20240501 20:08:15.670713 11729 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 20:08:15.670866 11729 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 20:08:15.674248 11729 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 20:08:15.674448 11729 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 20:08:15.678515 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:15.678885 11118 tablet_replica.cc:330] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc: stopping tablet replica
I20240501 20:08:15.679077 11118 raft_consensus.cc:2229] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.679302 11118 raft_consensus.cc:2258] T 6a4553e4778b4dbf85b2bbb5f855247a P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.679939 11118 tablet_replica.cc:330] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc: stopping tablet replica
I20240501 20:08:15.680127 11118 raft_consensus.cc:2229] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:15.680510 11118 raft_consensus.cc:2258] T ce9aa1175371401d8ca1db92b34cf504 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.681315 11118 tablet_replica.cc:330] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc: stopping tablet replica
I20240501 20:08:15.681484 11118 raft_consensus.cc:2229] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.681655 11118 raft_consensus.cc:2258] T 931bdc7b4a9e42d6a7a36d050424057e P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.682118 11118 tablet_replica.cc:330] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc: stopping tablet replica
I20240501 20:08:15.682286 11118 raft_consensus.cc:2229] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.682513 11118 raft_consensus.cc:2258] T e50657592d36432bac74f2c6a032ceae P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.682965 11118 tablet_replica.cc:330] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc: stopping tablet replica
I20240501 20:08:15.683117 11118 raft_consensus.cc:2229] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.683281 11118 raft_consensus.cc:2258] T e3c5b2965cd04377b0d24e9fededf0f1 P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.683715 11118 tablet_replica.cc:330] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc: stopping tablet replica
I20240501 20:08:15.683874 11118 raft_consensus.cc:2229] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:15.684226 11118 raft_consensus.cc:2258] T 4f6abd6198bb4e39b00e9e5c7dfcaeb6 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.684923 11118 tablet_replica.cc:330] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc: stopping tablet replica
I20240501 20:08:15.685145 11118 raft_consensus.cc:2229] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.685354 11118 raft_consensus.cc:2258] T ff74f06417f44eb59a118c69770ab6fa P ceb448c0562d444e8cbf5f256b20e9bc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.685973 11118 tablet_replica.cc:330] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc: stopping tablet replica
I20240501 20:08:15.686144 11118 raft_consensus.cc:2229] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:15.686355 11118 raft_consensus.cc:2258] T 493dbd49298242408e327fd183b13034 P ceb448c0562d444e8cbf5f256b20e9bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.702672 11118 tablet_server.cc:194] TabletServer@127.10.219.131:0 shutdown complete.
I20240501 20:08:15.707147 11118 tablet_server.cc:178] TabletServer@127.10.219.132:0 shutting down...
I20240501 20:08:15.715402 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:15.728978 11118 tablet_server.cc:194] TabletServer@127.10.219.132:0 shutdown complete.
I20240501 20:08:15.732132 11118 master.cc:537] Master@127.10.219.190:41825 shutting down...
I20240501 20:08:15.741632 11118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:15.741956 11118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:15.742085 11118 tablet_replica.cc:330] T 00000000000000000000000000000000 P 54e9aabab6d845698159a20b235b874d: stopping tablet replica
I20240501 20:08:15.746146 11118 master.cc:552] Master@127.10.219.190:41825 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7472 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240501 20:08:15.766795 11118 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.219.190:45253,127.10.219.189:38985,127.10.219.188:34647
I20240501 20:08:15.767490 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:15.771049 12154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:15.771507 12155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:15.772099 12157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:15.772852 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:15.773170 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:15.773244 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:15.773322 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594095773286 us; error 0 us; skew 500 ppm
I20240501 20:08:15.773510 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:15.775130 11118 webserver.cc:460] Webserver started at http://127.10.219.190:33601/ using document root <none> and password file <none>
I20240501 20:08:15.775398 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:15.775513 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:15.775696 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:15.776445 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-0-root/instance:
uuid: "44717f6de2034ea8b29e3e59136b0007"
format_stamp: "Formatted at 2024-05-01 20:08:15 on dist-test-slave-n6pz"
I20240501 20:08:15.779501 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 20:08:15.781536 12162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:15.782030 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240501 20:08:15.782164 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-0-root
uuid: "44717f6de2034ea8b29e3e59136b0007"
format_stamp: "Formatted at 2024-05-01 20:08:15 on dist-test-slave-n6pz"
I20240501 20:08:15.782666 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:15.791968 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:15.792704 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:15.807889 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.190:45253
I20240501 20:08:15.807920 12213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.190:45253 every 8 connection(s)
I20240501 20:08:15.809418 12214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:15.809545 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:08:15.812978 12214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } has no permanent_uuid. Determining permanent_uuid...
W20240501 20:08:15.813817 12217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:15.814460 12216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:15.815009 12219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:15.815804 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:15.816166 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:15.816260 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:15.816368 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594095816297 us; error 0 us; skew 500 ppm
I20240501 20:08:15.816567 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:15.818253 11118 webserver.cc:460] Webserver started at http://127.10.219.189:37045/ using document root <none> and password file <none>
I20240501 20:08:15.818585 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:15.818713 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:15.818868 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:15.819823 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-1-root/instance:
uuid: "697677b5f1ee40e3a188faaa7baea987"
format_stamp: "Formatted at 2024-05-01 20:08:15 on dist-test-slave-n6pz"
I20240501 20:08:15.821607 12214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:08:15.822968 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
W20240501 20:08:15.824177 12214 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.10.219.189:38985: Network error: Client connection negotiation failed: client connection to 127.10.219.189:38985: connect: Connection refused (error 111)
I20240501 20:08:15.824899 12227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:15.825242 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:08:15.825392 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-1-root
uuid: "697677b5f1ee40e3a188faaa7baea987"
format_stamp: "Formatted at 2024-05-01 20:08:15 on dist-test-slave-n6pz"
I20240501 20:08:15.825559 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:15.838799 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:15.839504 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:15.858166 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.189:38985
I20240501 20:08:15.858222 12278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.189:38985 every 8 connection(s)
I20240501 20:08:15.859992 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:08:15.860054 12279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:08:15.863749 12281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:15.864513 12282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:15.864552 12279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } has no permanent_uuid. Determining permanent_uuid...
W20240501 20:08:15.865181 12284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:15.865901 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:15.866204 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:15.866294 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:15.866428 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594095866382 us; error 0 us; skew 500 ppm
I20240501 20:08:15.866647 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:15.868497 11118 webserver.cc:460] Webserver started at http://127.10.219.188:40259/ using document root <none> and password file <none>
I20240501 20:08:15.868860 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:15.868981 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:15.869140 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:15.869925 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-2-root/instance:
uuid: "bc89180c1f4743b58c8739d717d5a62b"
format_stamp: "Formatted at 2024-05-01 20:08:15 on dist-test-slave-n6pz"
I20240501 20:08:15.872330 12279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:08:15.873297 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240501 20:08:15.875869 12291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:15.876341 11118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:08:15.876477 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-2-root
uuid: "bc89180c1f4743b58c8739d717d5a62b"
format_stamp: "Formatted at 2024-05-01 20:08:15 on dist-test-slave-n6pz"
I20240501 20:08:15.876663 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:15.881266 12279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } has no permanent_uuid. Determining permanent_uuid...
W20240501 20:08:15.883872 12279 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.10.219.188:34647: Network error: Client connection negotiation failed: client connection to 127.10.219.188:34647: connect: Connection refused (error 111)
I20240501 20:08:15.887545 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:15.888240 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:15.888463 12214 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } attempt: 1
I20240501 20:08:15.895896 12214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } has no permanent_uuid. Determining permanent_uuid...
W20240501 20:08:15.898449 12214 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.10.219.188:34647: Network error: Client connection negotiation failed: client connection to 127.10.219.188:34647: connect: Connection refused (error 111)
I20240501 20:08:15.905066 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.188:34647
I20240501 20:08:15.905100 12343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.188:34647 every 8 connection(s)
I20240501 20:08:15.906117 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:08:15.906507 12344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:15.909045 12344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:08:15.917804 12344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:08:15.924345 12344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:08:15.925732 12214 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } attempt: 1
I20240501 20:08:15.936056 12214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007: Bootstrap starting.
I20240501 20:08:15.936091 12344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b: Bootstrap starting.
I20240501 20:08:15.938397 12214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:15.938397 12344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:15.941084 12214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007: No bootstrap required, opened a new log
I20240501 20:08:15.941126 12344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b: No bootstrap required, opened a new log
I20240501 20:08:15.942150 12214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:15.942148 12344 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:15.942523 12214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:15.942548 12344 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:15.942654 12214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44717f6de2034ea8b29e3e59136b0007, State: Initialized, Role: FOLLOWER
I20240501 20:08:15.942657 12344 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc89180c1f4743b58c8739d717d5a62b, State: Initialized, Role: FOLLOWER
I20240501 20:08:15.943038 12344 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:15.943012 12214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:15.943854 12352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } } }
I20240501 20:08:15.944190 12352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:15.944206 12344 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:15.944206 12214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:15.943877 12351 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } } }
I20240501 20:08:15.945775 12351 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:15.949138 12279 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } attempt: 1
I20240501 20:08:15.954087 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240501 20:08:15.954391 12373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:08:15.954515 12373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 20:08:15.955881 12374 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:08:15.956030 12374 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:08:15.959703 12279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987: Bootstrap starting.
I20240501 20:08:15.961112 12279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:15.963114 12279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987: No bootstrap required, opened a new log
I20240501 20:08:15.963836 12279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:15.964047 12279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:15.964108 12279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697677b5f1ee40e3a188faaa7baea987, State: Initialized, Role: FOLLOWER
I20240501 20:08:15.964373 12279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:15.965154 12376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } } }
I20240501 20:08:15.965410 12376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:15.965533 12279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:15.970033 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240501 20:08:15.971381 12387 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:08:15.971547 12387 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:08:15.971668 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:15.976744 12388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:15.977210 12389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:15.977955 12391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:15.978899 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:15.979194 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:15.979277 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:15.979533 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594095979302 us; error 0 us; skew 500 ppm
I20240501 20:08:15.979749 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:15.981333 11118 webserver.cc:460] Webserver started at http://127.10.219.129:42751/ using document root <none> and password file <none>
I20240501 20:08:15.981601 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:15.981688 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:15.981814 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:15.982609 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-0-root/instance:
uuid: "16767a6cc7214442b4dcbccd6f05ae49"
format_stamp: "Formatted at 2024-05-01 20:08:15 on dist-test-slave-n6pz"
I20240501 20:08:15.985566 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:08:15.987625 12396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:15.988044 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 20:08:15.988160 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-0-root
uuid: "16767a6cc7214442b4dcbccd6f05ae49"
format_stamp: "Formatted at 2024-05-01 20:08:15 on dist-test-slave-n6pz"
I20240501 20:08:15.988324 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:16.006469 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:16.007217 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:16.007943 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:16.009450 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:16.009572 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:16.009711 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:16.009797 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:16.028460 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.129:39531
I20240501 20:08:16.028501 12458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.129:39531 every 8 connection(s)
I20240501 20:08:16.040627 12352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:16.040964 12352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:16.042161 12352 leader_election.cc:288] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 697677b5f1ee40e3a188faaa7baea987 (127.10.219.189:38985), bc89180c1f4743b58c8739d717d5a62b (127.10.219.188:34647)
I20240501 20:08:16.042938 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44717f6de2034ea8b29e3e59136b0007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "697677b5f1ee40e3a188faaa7baea987" is_pre_election: true
I20240501 20:08:16.042888 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44717f6de2034ea8b29e3e59136b0007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc89180c1f4743b58c8739d717d5a62b" is_pre_election: true
I20240501 20:08:16.043215 12459 heartbeater.cc:347] Connected to a master server at 127.10.219.188:34647
I20240501 20:08:16.043287 12254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44717f6de2034ea8b29e3e59136b0007 in term 0.
I20240501 20:08:16.043378 12459 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:16.043308 12319 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44717f6de2034ea8b29e3e59136b0007 in term 0.
I20240501 20:08:16.044035 12459 heartbeater.cc:511] Master 127.10.219.188:34647 requested a full tablet report, sending...
I20240501 20:08:16.044070 12164 leader_election.cc:302] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 44717f6de2034ea8b29e3e59136b0007, 697677b5f1ee40e3a188faaa7baea987, bc89180c1f4743b58c8739d717d5a62b; no voters:
I20240501 20:08:16.045004 12352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:16.045138 12352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:16.045269 12352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:16.048485 12352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:16.050374 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44717f6de2034ea8b29e3e59136b0007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "697677b5f1ee40e3a188faaa7baea987"
I20240501 20:08:16.050974 12254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:16.051254 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44717f6de2034ea8b29e3e59136b0007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc89180c1f4743b58c8739d717d5a62b"
I20240501 20:08:16.051704 12319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:16.054448 12319 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44717f6de2034ea8b29e3e59136b0007 in term 1.
I20240501 20:08:16.054487 12352 leader_election.cc:288] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [CANDIDATE]: Term 1 election: Requested vote from peers 697677b5f1ee40e3a188faaa7baea987 (127.10.219.189:38985), bc89180c1f4743b58c8739d717d5a62b (127.10.219.188:34647)
I20240501 20:08:16.055895 12163 leader_election.cc:302] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44717f6de2034ea8b29e3e59136b0007, bc89180c1f4743b58c8739d717d5a62b; no voters:
I20240501 20:08:16.056522 12309 ts_manager.cc:194] Registered new tserver with Master: 16767a6cc7214442b4dcbccd6f05ae49 (127.10.219.129:39531)
I20240501 20:08:16.057235 12352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:16.056762 12254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44717f6de2034ea8b29e3e59136b0007 in term 1.
I20240501 20:08:16.058221 12461 heartbeater.cc:347] Connected to a master server at 127.10.219.189:38985
I20240501 20:08:16.058508 12461 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:16.059008 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:08:16.059405 12461 heartbeater.cc:511] Master 127.10.219.189:38985 requested a full tablet report, sending...
I20240501 20:08:16.061244 12244 ts_manager.cc:194] Registered new tserver with Master: 16767a6cc7214442b4dcbccd6f05ae49 (127.10.219.129:39531)
I20240501 20:08:16.062582 12352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 1 LEADER]: Becoming Leader. State: Replica: 44717f6de2034ea8b29e3e59136b0007, State: Running, Role: LEADER
I20240501 20:08:16.063131 12352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:16.065140 12460 heartbeater.cc:347] Connected to a master server at 127.10.219.190:45253
I20240501 20:08:16.065428 12460 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:16.066298 12460 heartbeater.cc:511] Master 127.10.219.190:45253 requested a full tablet report, sending...
I20240501 20:08:16.069496 12466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44717f6de2034ea8b29e3e59136b0007. Latest consensus state: current_term: 1 leader_uuid: "44717f6de2034ea8b29e3e59136b0007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } } }
I20240501 20:08:16.070403 12179 ts_manager.cc:194] Registered new tserver with Master: 16767a6cc7214442b4dcbccd6f05ae49 (127.10.219.129:39531)
I20240501 20:08:16.070632 12466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [sys.catalog]: This master's current role is: LEADER
W20240501 20:08:16.075160 12468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:16.076200 12467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:16.078673 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:16.079308 12473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:16.081104 12473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20240501 20:08:16.081107 12470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:16.081845 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:16.081971 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:16.082060 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594096082018 us; error 0 us; skew 500 ppm
I20240501 20:08:16.082247 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:16.084378 11118 webserver.cc:460] Webserver started at http://127.10.219.130:37481/ using document root <none> and password file <none>
I20240501 20:08:16.084682 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:16.084827 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:16.085000 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:16.085894 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-1-root/instance:
uuid: "386f8265331645e48b164565968c9e4e"
format_stamp: "Formatted at 2024-05-01 20:08:16 on dist-test-slave-n6pz"
I20240501 20:08:16.086153 12254 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 1 FOLLOWER]: Refusing update from remote peer 44717f6de2034ea8b29e3e59136b0007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:16.086917 12319 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 1 FOLLOWER]: Refusing update from remote peer 44717f6de2034ea8b29e3e59136b0007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:16.087533 12466 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:16.087824 12352 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:16.097311 12376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44717f6de2034ea8b29e3e59136b0007. Latest consensus state: current_term: 1 leader_uuid: "44717f6de2034ea8b29e3e59136b0007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } } }
I20240501 20:08:16.097895 12376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:16.099220 12351 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44717f6de2034ea8b29e3e59136b0007. Latest consensus state: current_term: 1 leader_uuid: "44717f6de2034ea8b29e3e59136b0007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } } }
I20240501 20:08:16.099671 12351 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:16.102286 12473 catalog_manager.cc:1347] Generated new cluster ID: db9590f3082a4a1b925cfab3d76a9244
I20240501 20:08:16.102802 12473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:16.103928 12466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44717f6de2034ea8b29e3e59136b0007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } } }
I20240501 20:08:16.104364 12466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:16.104682 12466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44717f6de2034ea8b29e3e59136b0007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } } }
I20240501 20:08:16.105026 12466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:16.111733 12376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44717f6de2034ea8b29e3e59136b0007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } } }
I20240501 20:08:16.112246 12376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:16.115101 12351 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44717f6de2034ea8b29e3e59136b0007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } } }
I20240501 20:08:16.115476 12351 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:08:16.115854 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.030s user 0.008s sys 0.021s
I20240501 20:08:16.118126 12485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:16.118669 11118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:16.118820 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-1-root
uuid: "386f8265331645e48b164565968c9e4e"
format_stamp: "Formatted at 2024-05-01 20:08:16 on dist-test-slave-n6pz"
I20240501 20:08:16.118996 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:16.125816 12473 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:16.127211 12473 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:16.137228 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:16.138074 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:16.138938 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:16.141458 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:16.141575 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:16.141717 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:16.141791 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:16.144340 12473 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007: Generated new TSK 0
I20240501 20:08:16.144737 12473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:16.166544 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.130:45485
I20240501 20:08:16.166564 12548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.130:45485 every 8 connection(s)
I20240501 20:08:16.169765 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:16.183468 12556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:16.185413 12550 heartbeater.cc:347] Connected to a master server at 127.10.219.190:45253
I20240501 20:08:16.185657 12550 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:16.186414 12550 heartbeater.cc:511] Master 127.10.219.190:45253 requested a full tablet report, sending...
I20240501 20:08:16.187870 12179 ts_manager.cc:194] Registered new tserver with Master: 386f8265331645e48b164565968c9e4e (127.10.219.130:45485)
I20240501 20:08:16.190508 12179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60640
W20240501 20:08:16.193224 12557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:16.194106 12551 heartbeater.cc:347] Connected to a master server at 127.10.219.189:38985
I20240501 20:08:16.194300 12551 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:16.194401 11118 server_base.cc:1034] running on GCE node
W20240501 20:08:16.194790 12559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:16.194818 12551 heartbeater.cc:511] Master 127.10.219.189:38985 requested a full tablet report, sending...
I20240501 20:08:16.195294 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:16.195431 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:16.195509 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594096195481 us; error 0 us; skew 500 ppm
I20240501 20:08:16.195753 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:16.196240 12244 ts_manager.cc:194] Registered new tserver with Master: 386f8265331645e48b164565968c9e4e (127.10.219.130:45485)
I20240501 20:08:16.197279 12549 heartbeater.cc:347] Connected to a master server at 127.10.219.188:34647
I20240501 20:08:16.197523 12549 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:16.197629 11118 webserver.cc:460] Webserver started at http://127.10.219.131:45323/ using document root <none> and password file <none>
I20240501 20:08:16.197840 12549 heartbeater.cc:511] Master 127.10.219.188:34647 requested a full tablet report, sending...
I20240501 20:08:16.197938 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:16.198047 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:16.198233 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:16.199162 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-2-root/instance:
uuid: "684a92fb5a434e76826d1786168e2074"
format_stamp: "Formatted at 2024-05-01 20:08:16 on dist-test-slave-n6pz"
I20240501 20:08:16.199173 12309 ts_manager.cc:194] Registered new tserver with Master: 386f8265331645e48b164565968c9e4e (127.10.219.130:45485)
I20240501 20:08:16.202083 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:08:16.204156 12564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:16.204638 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240501 20:08:16.204785 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-2-root
uuid: "684a92fb5a434e76826d1786168e2074"
format_stamp: "Formatted at 2024-05-01 20:08:16 on dist-test-slave-n6pz"
I20240501 20:08:16.204929 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714594084913313-11118-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:16.218128 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:16.218956 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:16.219686 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:16.220762 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:16.220868 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:16.220980 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:16.221063 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:16.241215 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.131:38927
I20240501 20:08:16.241313 12626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.131:38927 every 8 connection(s)
I20240501 20:08:16.252497 12627 heartbeater.cc:347] Connected to a master server at 127.10.219.188:34647
I20240501 20:08:16.252710 12627 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:16.253476 12627 heartbeater.cc:511] Master 127.10.219.188:34647 requested a full tablet report, sending...
I20240501 20:08:16.254426 12629 heartbeater.cc:347] Connected to a master server at 127.10.219.189:38985
I20240501 20:08:16.254568 12629 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:16.254958 12309 ts_manager.cc:194] Registered new tserver with Master: 684a92fb5a434e76826d1786168e2074 (127.10.219.131:38927)
I20240501 20:08:16.255416 12629 heartbeater.cc:511] Master 127.10.219.189:38985 requested a full tablet report, sending...
I20240501 20:08:16.256630 12244 ts_manager.cc:194] Registered new tserver with Master: 684a92fb5a434e76826d1786168e2074 (127.10.219.131:38927)
I20240501 20:08:16.257166 12628 heartbeater.cc:347] Connected to a master server at 127.10.219.190:45253
I20240501 20:08:16.257287 12628 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:16.257711 12628 heartbeater.cc:511] Master 127.10.219.190:45253 requested a full tablet report, sending...
I20240501 20:08:16.258601 12179 ts_manager.cc:194] Registered new tserver with Master: 684a92fb5a434e76826d1786168e2074 (127.10.219.131:38927)
I20240501 20:08:16.259495 12179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60650
I20240501 20:08:16.259716 11118 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015583863s
I20240501 20:08:16.956063 12373 catalog_manager.cc:1259] Loaded cluster ID: db9590f3082a4a1b925cfab3d76a9244
I20240501 20:08:16.956205 12373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b: loading cluster ID for follower catalog manager: success
I20240501 20:08:16.958245 12373 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b: acquiring CA information for follower catalog manager: success
I20240501 20:08:16.959432 12373 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 20:08:16.973093 12387 catalog_manager.cc:1259] Loaded cluster ID: db9590f3082a4a1b925cfab3d76a9244
I20240501 20:08:16.973229 12387 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987: loading cluster ID for follower catalog manager: success
I20240501 20:08:16.975204 12387 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987: acquiring CA information for follower catalog manager: success
I20240501 20:08:16.976540 12387 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 20:08:17.074079 12179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60628
I20240501 20:08:17.192798 12550 heartbeater.cc:503] Master 127.10.219.190:45253 was elected leader, sending a full tablet report...
I20240501 20:08:17.261301 12628 heartbeater.cc:503] Master 127.10.219.190:45253 was elected leader, sending a full tablet report...
I20240501 20:08:17.952523 12372 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:18.075726 12460 heartbeater.cc:503] Master 127.10.219.190:45253 was elected leader, sending a full tablet report...
I20240501 20:08:18.316798 11118 test_util.cc:268] Using random seed: 204080616
I20240501 20:08:18.340803 12179 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34474:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 20:08:18.371335 12424 tablet_service.cc:1467] Processing CreateTablet for tablet d1d48f210c4143678f5676c6e52fb201 (DEFAULT_TABLE table=test-workload [id=085887c00bdd454c9e8aa917cb256a85]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 20:08:18.371500 12514 tablet_service.cc:1467] Processing CreateTablet for tablet 8cea2e7c95ff4d26aacd811ebee4a300 (DEFAULT_TABLE table=test-workload [id=085887c00bdd454c9e8aa917cb256a85]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 20:08:18.372018 12514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cea2e7c95ff4d26aacd811ebee4a300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:18.372051 12424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d48f210c4143678f5676c6e52fb201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:18.376556 12592 tablet_service.cc:1467] Processing CreateTablet for tablet 08b6a08075114a429aa75aeb286e670d (DEFAULT_TABLE table=test-workload [id=085887c00bdd454c9e8aa917cb256a85]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 20:08:18.377342 12592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b6a08075114a429aa75aeb286e670d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:18.379259 12663 tablet_bootstrap.cc:492] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49: Bootstrap starting.
I20240501 20:08:18.380043 12664 tablet_bootstrap.cc:492] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e: Bootstrap starting.
I20240501 20:08:18.382555 12663 tablet_bootstrap.cc:654] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:18.382602 12664 tablet_bootstrap.cc:654] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:18.385915 12663 tablet_bootstrap.cc:492] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49: No bootstrap required, opened a new log
I20240501 20:08:18.385952 12664 tablet_bootstrap.cc:492] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e: No bootstrap required, opened a new log
I20240501 20:08:18.386029 12667 tablet_bootstrap.cc:492] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074: Bootstrap starting.
I20240501 20:08:18.386505 12663 ts_tablet_manager.cc:1367] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240501 20:08:18.386554 12664 ts_tablet_manager.cc:1367] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240501 20:08:18.387542 12664 raft_consensus.cc:348] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "386f8265331645e48b164565968c9e4e" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 45485 } }
I20240501 20:08:18.387737 12663 raft_consensus.cc:348] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16767a6cc7214442b4dcbccd6f05ae49" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 39531 } }
I20240501 20:08:18.387753 12664 raft_consensus.cc:374] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:18.388003 12663 raft_consensus.cc:374] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:18.388058 12664 raft_consensus.cc:729] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386f8265331645e48b164565968c9e4e, State: Initialized, Role: FOLLOWER
I20240501 20:08:18.388109 12663 raft_consensus.cc:729] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16767a6cc7214442b4dcbccd6f05ae49, State: Initialized, Role: FOLLOWER
I20240501 20:08:18.388305 12664 consensus_queue.cc:260] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "386f8265331645e48b164565968c9e4e" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 45485 } }
I20240501 20:08:18.388365 12663 consensus_queue.cc:260] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16767a6cc7214442b4dcbccd6f05ae49" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 39531 } }
I20240501 20:08:18.388482 12664 raft_consensus.cc:388] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:18.388562 12664 raft_consensus.cc:482] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:18.388568 12663 raft_consensus.cc:388] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:18.388736 12664 raft_consensus.cc:3037] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:18.388747 12663 raft_consensus.cc:482] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:18.388944 12663 raft_consensus.cc:3037] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:18.388988 12667 tablet_bootstrap.cc:654] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:18.391237 12664 raft_consensus.cc:504] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "386f8265331645e48b164565968c9e4e" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 45485 } }
I20240501 20:08:18.391549 12664 leader_election.cc:302] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 386f8265331645e48b164565968c9e4e; no voters:
I20240501 20:08:18.392069 12664 leader_election.cc:288] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:18.392387 12669 raft_consensus.cc:2781] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:18.392980 12663 raft_consensus.cc:504] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16767a6cc7214442b4dcbccd6f05ae49" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 39531 } }
I20240501 20:08:18.393452 12663 leader_election.cc:302] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16767a6cc7214442b4dcbccd6f05ae49; no voters:
I20240501 20:08:18.398602 12667 tablet_bootstrap.cc:492] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074: No bootstrap required, opened a new log
I20240501 20:08:18.398993 12667 ts_tablet_manager.cc:1367] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20240501 20:08:18.399617 12669 raft_consensus.cc:686] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [term 1 LEADER]: Becoming Leader. State: Replica: 386f8265331645e48b164565968c9e4e, State: Running, Role: LEADER
I20240501 20:08:18.400271 12669 consensus_queue.cc:237] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "386f8265331645e48b164565968c9e4e" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 45485 } }
I20240501 20:08:18.400794 12664 ts_tablet_manager.cc:1398] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20240501 20:08:18.400954 12663 leader_election.cc:288] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:18.401132 12672 raft_consensus.cc:2781] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:18.400434 12667 raft_consensus.cc:348] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684a92fb5a434e76826d1786168e2074" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 38927 } }
I20240501 20:08:18.401813 12667 raft_consensus.cc:374] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:18.402036 12667 raft_consensus.cc:729] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684a92fb5a434e76826d1786168e2074, State: Initialized, Role: FOLLOWER
I20240501 20:08:18.402618 12667 consensus_queue.cc:260] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684a92fb5a434e76826d1786168e2074" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 38927 } }
I20240501 20:08:18.403077 12667 raft_consensus.cc:388] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:18.403227 12667 raft_consensus.cc:482] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:18.403439 12667 raft_consensus.cc:3037] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:18.405474 12179 catalog_manager.cc:5484] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 386f8265331645e48b164565968c9e4e (127.10.219.130). New cstate: current_term: 1 leader_uuid: "386f8265331645e48b164565968c9e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "386f8265331645e48b164565968c9e4e" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 45485 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:18.409869 12663 ts_tablet_manager.cc:1398] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49: Time spent starting tablet: real 0.023s user 0.012s sys 0.007s
I20240501 20:08:18.410445 12672 raft_consensus.cc:686] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [term 1 LEADER]: Becoming Leader. State: Replica: 16767a6cc7214442b4dcbccd6f05ae49, State: Running, Role: LEADER
I20240501 20:08:18.410815 12667 raft_consensus.cc:504] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684a92fb5a434e76826d1786168e2074" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 38927 } }
I20240501 20:08:18.411221 12667 leader_election.cc:302] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 684a92fb5a434e76826d1786168e2074; no voters:
I20240501 20:08:18.410904 12672 consensus_queue.cc:237] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16767a6cc7214442b4dcbccd6f05ae49" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 39531 } }
I20240501 20:08:18.421125 12179 catalog_manager.cc:5484] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16767a6cc7214442b4dcbccd6f05ae49 (127.10.219.129). New cstate: current_term: 1 leader_uuid: "16767a6cc7214442b4dcbccd6f05ae49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16767a6cc7214442b4dcbccd6f05ae49" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 39531 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:18.422574 12667 leader_election.cc:288] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:18.419831 12677 raft_consensus.cc:2781] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:18.423568 12667 ts_tablet_manager.cc:1398] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074: Time spent starting tablet: real 0.024s user 0.008s sys 0.006s
I20240501 20:08:18.424495 12677 raft_consensus.cc:686] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [term 1 LEADER]: Becoming Leader. State: Replica: 684a92fb5a434e76826d1786168e2074, State: Running, Role: LEADER
I20240501 20:08:18.424870 12677 consensus_queue.cc:237] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684a92fb5a434e76826d1786168e2074" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 38927 } }
I20240501 20:08:18.436354 12178 catalog_manager.cc:5484] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 reported cstate change: term changed from 0 to 1, leader changed from <none> to 684a92fb5a434e76826d1786168e2074 (127.10.219.131). New cstate: current_term: 1 leader_uuid: "684a92fb5a434e76826d1786168e2074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684a92fb5a434e76826d1786168e2074" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 38927 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:18.444612 11118 master.cc:537] Master@127.10.219.190:45253 shutting down...
W20240501 20:08:18.449944 12628 heartbeater.cc:644] Failed to heartbeat to 127.10.219.190:45253 (0 consecutive failures): Remote error: Failed to send heartbeat to master: Service unavailable: service kudu.master.MasterService not registered on Master
I20240501 20:08:18.452968 11118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:18.453351 11118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:18.453469 11118 tablet_replica.cc:330] T 00000000000000000000000000000000 P 44717f6de2034ea8b29e3e59136b0007: stopping tablet replica
I20240501 20:08:18.471505 11118 master.cc:552] Master@127.10.219.190:45253 shutdown complete.
I20240501 20:08:19.943953 12687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44717f6de2034ea8b29e3e59136b0007)
I20240501 20:08:19.944031 12688 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44717f6de2034ea8b29e3e59136b0007)
I20240501 20:08:19.944228 12688 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:19.944195 12687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:19.944975 12687 leader_election.cc:288] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44717f6de2034ea8b29e3e59136b0007 (127.10.219.190:45253), bc89180c1f4743b58c8739d717d5a62b (127.10.219.188:34647)
I20240501 20:08:19.945117 12688 leader_election.cc:288] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44717f6de2034ea8b29e3e59136b0007 (127.10.219.190:45253), 697677b5f1ee40e3a188faaa7baea987 (127.10.219.189:38985)
I20240501 20:08:19.946203 12253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc89180c1f4743b58c8739d717d5a62b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "697677b5f1ee40e3a188faaa7baea987" is_pre_election: true
I20240501 20:08:19.946540 12253 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc89180c1f4743b58c8739d717d5a62b in term 1.
I20240501 20:08:19.948158 12293 leader_election.cc:302] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 697677b5f1ee40e3a188faaa7baea987, bc89180c1f4743b58c8739d717d5a62b; no voters:
I20240501 20:08:19.948393 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "697677b5f1ee40e3a188faaa7baea987" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "bc89180c1f4743b58c8739d717d5a62b" is_pre_election: true
I20240501 20:08:19.948577 12688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:08:19.948706 12319 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 697677b5f1ee40e3a188faaa7baea987 in term 1.
I20240501 20:08:19.949079 12688 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44717f6de2034ea8b29e3e59136b0007)
I20240501 20:08:19.949214 12688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:19.950109 12228 leader_election.cc:302] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 697677b5f1ee40e3a188faaa7baea987, bc89180c1f4743b58c8739d717d5a62b; no voters:
W20240501 20:08:19.950409 12292 leader_election.cc:334] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 44717f6de2034ea8b29e3e59136b0007 (127.10.219.190:45253): Network error: Client connection negotiation failed: client connection to 127.10.219.190:45253: connect: Connection refused (error 111)
I20240501 20:08:19.950824 12687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:08:19.950968 12687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44717f6de2034ea8b29e3e59136b0007)
I20240501 20:08:19.951084 12687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:08:19.951339 12228 leader_election.cc:334] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 44717f6de2034ea8b29e3e59136b0007 (127.10.219.190:45253): Network error: Client connection negotiation failed: client connection to 127.10.219.190:45253: connect: Connection refused (error 111)
I20240501 20:08:19.952291 12688 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:19.953130 12688 leader_election.cc:288] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [CANDIDATE]: Term 2 election: Requested vote from peers 44717f6de2034ea8b29e3e59136b0007 (127.10.219.190:45253), 697677b5f1ee40e3a188faaa7baea987 (127.10.219.189:38985)
I20240501 20:08:19.953779 12253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc89180c1f4743b58c8739d717d5a62b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "697677b5f1ee40e3a188faaa7baea987"
I20240501 20:08:19.953902 12687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:19.954365 12253 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bc89180c1f4743b58c8739d717d5a62b in current term 2: Already voted for candidate 697677b5f1ee40e3a188faaa7baea987 in this term.
I20240501 20:08:19.954602 12687 leader_election.cc:288] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [CANDIDATE]: Term 2 election: Requested vote from peers 44717f6de2034ea8b29e3e59136b0007 (127.10.219.190:45253), bc89180c1f4743b58c8739d717d5a62b (127.10.219.188:34647)
W20240501 20:08:19.954942 12292 leader_election.cc:334] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 44717f6de2034ea8b29e3e59136b0007 (127.10.219.190:45253): Network error: Client connection negotiation failed: client connection to 127.10.219.190:45253: connect: Connection refused (error 111)
I20240501 20:08:19.955104 12292 leader_election.cc:302] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc89180c1f4743b58c8739d717d5a62b; no voters: 44717f6de2034ea8b29e3e59136b0007, 697677b5f1ee40e3a188faaa7baea987
I20240501 20:08:19.955410 12688 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:08:19.956164 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "697677b5f1ee40e3a188faaa7baea987" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "bc89180c1f4743b58c8739d717d5a62b"
I20240501 20:08:19.956481 12319 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 697677b5f1ee40e3a188faaa7baea987 in current term 2: Already voted for candidate bc89180c1f4743b58c8739d717d5a62b in this term.
W20240501 20:08:19.956831 12228 leader_election.cc:334] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 44717f6de2034ea8b29e3e59136b0007 (127.10.219.190:45253): Network error: Client connection negotiation failed: client connection to 127.10.219.190:45253: connect: Connection refused (error 111)
I20240501 20:08:19.957084 12228 leader_election.cc:302] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 697677b5f1ee40e3a188faaa7baea987; no voters: 44717f6de2034ea8b29e3e59136b0007, bc89180c1f4743b58c8739d717d5a62b
I20240501 20:08:19.957345 12687 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:08:21.833622 12699 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:21.833927 12699 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:21.834758 12699 leader_election.cc:288] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 44717f6de2034ea8b29e3e59136b0007 (127.10.219.190:45253), 697677b5f1ee40e3a188faaa7baea987 (127.10.219.189:38985)
I20240501 20:08:21.835337 12253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc89180c1f4743b58c8739d717d5a62b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "697677b5f1ee40e3a188faaa7baea987" is_pre_election: true
I20240501 20:08:21.835588 12253 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc89180c1f4743b58c8739d717d5a62b in term 2.
W20240501 20:08:21.835810 12292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.219.190:45253: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 20:08:21.836071 12293 leader_election.cc:302] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 697677b5f1ee40e3a188faaa7baea987, bc89180c1f4743b58c8739d717d5a62b; no voters:
I20240501 20:08:21.836426 12699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:08:21.836532 12699 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:21.836652 12699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:08:21.837220 12292 leader_election.cc:334] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 44717f6de2034ea8b29e3e59136b0007 (127.10.219.190:45253): Network error: Client connection negotiation failed: client connection to 127.10.219.190:45253: connect: Connection refused (error 111)
I20240501 20:08:21.839010 12699 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:21.839708 12699 leader_election.cc:288] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [CANDIDATE]: Term 3 election: Requested vote from peers 44717f6de2034ea8b29e3e59136b0007 (127.10.219.190:45253), 697677b5f1ee40e3a188faaa7baea987 (127.10.219.189:38985)
I20240501 20:08:21.840351 12253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc89180c1f4743b58c8739d717d5a62b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "697677b5f1ee40e3a188faaa7baea987"
I20240501 20:08:21.840569 12253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:08:21.841084 12292 leader_election.cc:334] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 44717f6de2034ea8b29e3e59136b0007 (127.10.219.190:45253): Network error: Client connection negotiation failed: client connection to 127.10.219.190:45253: connect: Connection refused (error 111)
I20240501 20:08:21.842902 12253 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc89180c1f4743b58c8739d717d5a62b in term 3.
I20240501 20:08:21.843398 12293 leader_election.cc:302] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 697677b5f1ee40e3a188faaa7baea987, bc89180c1f4743b58c8739d717d5a62b; no voters: 44717f6de2034ea8b29e3e59136b0007
I20240501 20:08:21.843751 12699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:08:21.844168 12699 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 3 LEADER]: Becoming Leader. State: Replica: bc89180c1f4743b58c8739d717d5a62b, State: Running, Role: LEADER
I20240501 20:08:21.844444 12699 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } }
I20240501 20:08:21.845866 12702 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc89180c1f4743b58c8739d717d5a62b. Latest consensus state: current_term: 3 leader_uuid: "bc89180c1f4743b58c8739d717d5a62b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44717f6de2034ea8b29e3e59136b0007" member_type: VOTER last_known_addr { host: "127.10.219.190" port: 45253 } } peers { permanent_uuid: "697677b5f1ee40e3a188faaa7baea987" member_type: VOTER last_known_addr { host: "127.10.219.189" port: 38985 } } peers { permanent_uuid: "bc89180c1f4743b58c8739d717d5a62b" member_type: VOTER last_known_addr { host: "127.10.219.188" port: 34647 } } }
I20240501 20:08:21.846112 12702 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:21.846786 12704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:21.848896 12704 catalog_manager.cc:669] Loaded metadata for table test-workload [id=085887c00bdd454c9e8aa917cb256a85]
I20240501 20:08:21.850700 12704 tablet_loader.cc:96] loaded metadata for tablet 08b6a08075114a429aa75aeb286e670d (table test-workload [id=085887c00bdd454c9e8aa917cb256a85])
I20240501 20:08:21.851140 12704 tablet_loader.cc:96] loaded metadata for tablet 8cea2e7c95ff4d26aacd811ebee4a300 (table test-workload [id=085887c00bdd454c9e8aa917cb256a85])
I20240501 20:08:21.851509 12704 tablet_loader.cc:96] loaded metadata for tablet d1d48f210c4143678f5676c6e52fb201 (table test-workload [id=085887c00bdd454c9e8aa917cb256a85])
I20240501 20:08:21.851790 12704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:21.852871 12704 catalog_manager.cc:1259] Loaded cluster ID: db9590f3082a4a1b925cfab3d76a9244
I20240501 20:08:21.852985 12704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:21.854581 12704 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:21.855963 12704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b: Loaded TSK: 0
I20240501 20:08:21.856820 12704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:21.953222 12371 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:21.953397 12371 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:21.963950 11118 tablet_server.cc:178] TabletServer@127.10.219.129:0 shutting down...
I20240501 20:08:21.975777 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:21.976337 11118 tablet_replica.cc:330] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49: stopping tablet replica
I20240501 20:08:21.976583 11118 raft_consensus.cc:2229] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:21.976887 11118 raft_consensus.cc:2258] T d1d48f210c4143678f5676c6e52fb201 P 16767a6cc7214442b4dcbccd6f05ae49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:21.992285 11118 tablet_server.cc:194] TabletServer@127.10.219.129:0 shutdown complete.
I20240501 20:08:21.995959 11118 tablet_server.cc:178] TabletServer@127.10.219.130:0 shutting down...
I20240501 20:08:22.005880 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:22.006181 11118 tablet_replica.cc:330] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e: stopping tablet replica
I20240501 20:08:22.006459 11118 raft_consensus.cc:2229] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:22.006687 11118 raft_consensus.cc:2258] T 8cea2e7c95ff4d26aacd811ebee4a300 P 386f8265331645e48b164565968c9e4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:22.021268 11118 tablet_server.cc:194] TabletServer@127.10.219.130:0 shutdown complete.
I20240501 20:08:22.024930 11118 tablet_server.cc:178] TabletServer@127.10.219.131:0 shutting down...
I20240501 20:08:22.034046 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:22.034395 11118 tablet_replica.cc:330] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074: stopping tablet replica
I20240501 20:08:22.034606 11118 raft_consensus.cc:2229] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:22.034816 11118 raft_consensus.cc:2258] T 08b6a08075114a429aa75aeb286e670d P 684a92fb5a434e76826d1786168e2074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:22.048858 11118 tablet_server.cc:194] TabletServer@127.10.219.131:0 shutdown complete.
I20240501 20:08:22.052196 11118 master.cc:537] Master@127.10.219.189:38985 shutting down...
I20240501 20:08:22.060469 11118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:22.060750 11118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:22.060875 11118 tablet_replica.cc:330] T 00000000000000000000000000000000 P 697677b5f1ee40e3a188faaa7baea987: stopping tablet replica
I20240501 20:08:22.075107 11118 master.cc:552] Master@127.10.219.189:38985 shutdown complete.
I20240501 20:08:22.080168 11118 master.cc:537] Master@127.10.219.188:34647 shutting down...
I20240501 20:08:22.088721 11118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 3 LEADER]: Raft consensus shutting down.
I20240501 20:08:22.089118 11118 pending_rounds.cc:62] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b: Trying to abort 1 pending ops.
I20240501 20:08:22.089268 11118 pending_rounds.cc:68] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b: Aborting op as it isn't in flight: id { term: 3 index: 11 } timestamp: 7022977441157685248 op_type: NO_OP noop_request { }
I20240501 20:08:22.089476 11118 raft_consensus.cc:2866] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:08:22.089612 11118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:22.089838 11118 tablet_replica.cc:330] T 00000000000000000000000000000000 P bc89180c1f4743b58c8739d717d5a62b: stopping tablet replica
I20240501 20:08:22.096165 11118 master.cc:552] Master@127.10.219.188:34647 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6344 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240501 20:08:22.111351 11118 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.219.190:42981
I20240501 20:08:22.111984 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:22.115487 12705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:22.116052 12706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:22.116856 12708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:22.117630 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:22.117913 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:22.117990 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:22.118036 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594102118021 us; error 0 us; skew 500 ppm
I20240501 20:08:22.118204 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:22.124933 11118 webserver.cc:460] Webserver started at http://127.10.219.190:34405/ using document root <none> and password file <none>
I20240501 20:08:22.125227 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:22.125304 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:22.125454 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:22.126224 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/master-0-root/instance:
uuid: "9946bfe043974779b46a1c4f8ae0da9d"
format_stamp: "Formatted at 2024-05-01 20:08:22 on dist-test-slave-n6pz"
I20240501 20:08:22.129386 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:22.131417 12713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:22.131829 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 20:08:22.131978 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/master-0-root
uuid: "9946bfe043974779b46a1c4f8ae0da9d"
format_stamp: "Formatted at 2024-05-01 20:08:22 on dist-test-slave-n6pz"
I20240501 20:08:22.132120 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:22.147935 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:22.148679 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:22.163352 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.190:42981
I20240501 20:08:22.163367 12764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.190:42981 every 8 connection(s)
I20240501 20:08:22.164916 12765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:22.169364 12765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d: Bootstrap starting.
I20240501 20:08:22.170863 12765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:22.172720 12765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d: No bootstrap required, opened a new log
I20240501 20:08:22.173458 12765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9946bfe043974779b46a1c4f8ae0da9d" member_type: VOTER }
I20240501 20:08:22.173821 12765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:22.174063 12765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9946bfe043974779b46a1c4f8ae0da9d, State: Initialized, Role: FOLLOWER
I20240501 20:08:22.174564 12765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9946bfe043974779b46a1c4f8ae0da9d" member_type: VOTER }
I20240501 20:08:22.174815 12765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:22.174919 12765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:22.175057 12765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:22.177196 12765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9946bfe043974779b46a1c4f8ae0da9d" member_type: VOTER }
I20240501 20:08:22.177515 12765 leader_election.cc:302] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9946bfe043974779b46a1c4f8ae0da9d; no voters:
I20240501 20:08:22.178069 12765 leader_election.cc:288] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:22.178233 12768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:22.178778 12768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [term 1 LEADER]: Becoming Leader. State: Replica: 9946bfe043974779b46a1c4f8ae0da9d, State: Running, Role: LEADER
I20240501 20:08:22.179096 12768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9946bfe043974779b46a1c4f8ae0da9d" member_type: VOTER }
I20240501 20:08:22.179260 12765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:22.180191 12769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9946bfe043974779b46a1c4f8ae0da9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9946bfe043974779b46a1c4f8ae0da9d" member_type: VOTER } }
I20240501 20:08:22.180274 12770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9946bfe043974779b46a1c4f8ae0da9d. Latest consensus state: current_term: 1 leader_uuid: "9946bfe043974779b46a1c4f8ae0da9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9946bfe043974779b46a1c4f8ae0da9d" member_type: VOTER } }
I20240501 20:08:22.180687 12770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:22.180781 12769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:22.182008 12773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:22.183652 12773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:22.186117 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:08:22.187186 12773 catalog_manager.cc:1347] Generated new cluster ID: ef878903aef34bbb8bf938cac5011130
I20240501 20:08:22.187301 12773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:22.201979 12773 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:22.202924 12773 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:22.211165 12773 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d: Generated new TSK 0
I20240501 20:08:22.211516 12773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:22.219532 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:22.223286 12786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:22.223908 12787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:22.224695 12789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:22.225404 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:22.225711 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:22.225782 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:22.225867 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594102225826 us; error 0 us; skew 500 ppm
I20240501 20:08:22.226040 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:22.227617 11118 webserver.cc:460] Webserver started at http://127.10.219.129:46839/ using document root <none> and password file <none>
I20240501 20:08:22.227900 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:22.228014 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:22.228166 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:22.228937 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-0-root/instance:
uuid: "339dfd56e20342c1b0c3d7efb65a624b"
format_stamp: "Formatted at 2024-05-01 20:08:22 on dist-test-slave-n6pz"
I20240501 20:08:22.231935 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:22.233929 12794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:22.234423 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:08:22.234532 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-0-root
uuid: "339dfd56e20342c1b0c3d7efb65a624b"
format_stamp: "Formatted at 2024-05-01 20:08:22 on dist-test-slave-n6pz"
I20240501 20:08:22.234674 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:22.265542 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:22.266224 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:22.266889 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:22.267879 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:22.267949 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:22.268044 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:22.268112 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:22.283231 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.129:33071
I20240501 20:08:22.283277 12856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.129:33071 every 8 connection(s)
I20240501 20:08:22.285130 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:22.289469 12861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:22.290155 12862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:22.290953 12864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:22.291589 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:22.291913 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:22.292016 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:22.292091 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594102292067 us; error 0 us; skew 500 ppm
I20240501 20:08:22.292304 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:22.293736 12857 heartbeater.cc:347] Connected to a master server at 127.10.219.190:42981
I20240501 20:08:22.293892 12857 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:22.294349 12857 heartbeater.cc:511] Master 127.10.219.190:42981 requested a full tablet report, sending...
I20240501 20:08:22.294860 11118 webserver.cc:460] Webserver started at http://127.10.219.130:36559/ using document root <none> and password file <none>
I20240501 20:08:22.295169 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:22.295291 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:22.295491 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:22.295516 12730 ts_manager.cc:194] Registered new tserver with Master: 339dfd56e20342c1b0c3d7efb65a624b (127.10.219.129:33071)
I20240501 20:08:22.296437 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-1-root/instance:
uuid: "d388ae45a29d4a15af1448cddce316af"
format_stamp: "Formatted at 2024-05-01 20:08:22 on dist-test-slave-n6pz"
I20240501 20:08:22.297216 12730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39174
I20240501 20:08:22.299753 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 20:08:22.301577 12869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:22.301972 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:08:22.302114 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-1-root
uuid: "d388ae45a29d4a15af1448cddce316af"
format_stamp: "Formatted at 2024-05-01 20:08:22 on dist-test-slave-n6pz"
I20240501 20:08:22.302248 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:22.326479 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:22.327064 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:22.327616 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:22.328475 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:22.328534 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:22.328619 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:22.328663 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:22.342353 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.130:36327
I20240501 20:08:22.342386 12931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.130:36327 every 8 connection(s)
I20240501 20:08:22.344069 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:22.347635 12935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:22.348371 12936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:22.349113 12938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:22.349747 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:22.350031 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:22.350090 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:22.350129 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594102350124 us; error 0 us; skew 500 ppm
I20240501 20:08:22.350271 12932 heartbeater.cc:347] Connected to a master server at 127.10.219.190:42981
I20240501 20:08:22.350297 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:22.350445 12932 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:22.350833 12932 heartbeater.cc:511] Master 127.10.219.190:42981 requested a full tablet report, sending...
I20240501 20:08:22.351847 12730 ts_manager.cc:194] Registered new tserver with Master: d388ae45a29d4a15af1448cddce316af (127.10.219.130:36327)
I20240501 20:08:22.351863 11118 webserver.cc:460] Webserver started at http://127.10.219.131:37715/ using document root <none> and password file <none>
I20240501 20:08:22.352118 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:22.352196 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:22.352319 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:22.352748 12730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39182
I20240501 20:08:22.352988 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-2-root/instance:
uuid: "33d25e4ea3264248b12ddfc599f5c718"
format_stamp: "Formatted at 2024-05-01 20:08:22 on dist-test-slave-n6pz"
I20240501 20:08:22.355571 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:08:22.357167 12943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:22.357523 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 20:08:22.357641 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-2-root
uuid: "33d25e4ea3264248b12ddfc599f5c718"
format_stamp: "Formatted at 2024-05-01 20:08:22 on dist-test-slave-n6pz"
I20240501 20:08:22.357754 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:22.382987 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:22.383641 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:22.384208 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:22.385190 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:22.385257 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:22.385340 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:22.385381 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:22.400375 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.131:33839
I20240501 20:08:22.400411 13005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.131:33839 every 8 connection(s)
I20240501 20:08:22.407454 13006 heartbeater.cc:347] Connected to a master server at 127.10.219.190:42981
I20240501 20:08:22.407608 13006 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:22.407945 13006 heartbeater.cc:511] Master 127.10.219.190:42981 requested a full tablet report, sending...
I20240501 20:08:22.408859 12730 ts_manager.cc:194] Registered new tserver with Master: 33d25e4ea3264248b12ddfc599f5c718 (127.10.219.131:33839)
I20240501 20:08:22.409240 11118 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007920633s
I20240501 20:08:22.409713 12730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39188
I20240501 20:08:23.298795 12857 heartbeater.cc:503] Master 127.10.219.190:42981 was elected leader, sending a full tablet report...
I20240501 20:08:23.354054 12932 heartbeater.cc:503] Master 127.10.219.190:42981 was elected leader, sending a full tablet report...
I20240501 20:08:23.411211 13006 heartbeater.cc:503] Master 127.10.219.190:42981 was elected leader, sending a full tablet report...
I20240501 20:08:23.438087 11118 test_util.cc:268] Using random seed: 209201916
I20240501 20:08:23.451732 12730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39190:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:08:23.452486 12730 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:08:23.480677 12971 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbbd45dd08144adbbd51cad42bad500 (DEFAULT_TABLE table=test-workload [id=64c77da340c7495982e2991f98e5414f]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 20:08:23.480677 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 2166fbc27b124ebdb0b8ab3a6808a19d (DEFAULT_TABLE table=test-workload [id=64c77da340c7495982e2991f98e5414f]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 20:08:23.481467 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbbd45dd08144adbbd51cad42bad500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:23.481823 12822 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbbd45dd08144adbbd51cad42bad500 (DEFAULT_TABLE table=test-workload [id=64c77da340c7495982e2991f98e5414f]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 20:08:23.482380 12897 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbbd45dd08144adbbd51cad42bad500 (DEFAULT_TABLE table=test-workload [id=64c77da340c7495982e2991f98e5414f]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 20:08:23.482506 12896 tablet_service.cc:1467] Processing CreateTablet for tablet 2166fbc27b124ebdb0b8ab3a6808a19d (DEFAULT_TABLE table=test-workload [id=64c77da340c7495982e2991f98e5414f]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 20:08:23.482893 12897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbbd45dd08144adbbd51cad42bad500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:23.483139 12896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2166fbc27b124ebdb0b8ab3a6808a19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:23.483060 12821 tablet_service.cc:1467] Processing CreateTablet for tablet 2166fbc27b124ebdb0b8ab3a6808a19d (DEFAULT_TABLE table=test-workload [id=64c77da340c7495982e2991f98e5414f]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 20:08:23.483474 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2166fbc27b124ebdb0b8ab3a6808a19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:23.483943 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2166fbc27b124ebdb0b8ab3a6808a19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:23.484658 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbbd45dd08144adbbd51cad42bad500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:23.496662 13027 tablet_bootstrap.cc:492] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718: Bootstrap starting.
I20240501 20:08:23.497195 13026 tablet_bootstrap.cc:492] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af: Bootstrap starting.
I20240501 20:08:23.497933 13028 tablet_bootstrap.cc:492] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b: Bootstrap starting.
I20240501 20:08:23.499125 13027 tablet_bootstrap.cc:654] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:23.499477 13026 tablet_bootstrap.cc:654] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:23.499977 13028 tablet_bootstrap.cc:654] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:23.505719 13026 tablet_bootstrap.cc:492] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af: No bootstrap required, opened a new log
I20240501 20:08:23.505806 13027 tablet_bootstrap.cc:492] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718: No bootstrap required, opened a new log
I20240501 20:08:23.505928 13026 ts_tablet_manager.cc:1367] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240501 20:08:23.505928 13027 ts_tablet_manager.cc:1367] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240501 20:08:23.506295 13028 tablet_bootstrap.cc:492] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b: No bootstrap required, opened a new log
I20240501 20:08:23.506506 13028 ts_tablet_manager.cc:1367] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.008s
I20240501 20:08:23.506865 13026 raft_consensus.cc:348] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } }
I20240501 20:08:23.506858 13027 raft_consensus.cc:348] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } }
I20240501 20:08:23.507143 13027 raft_consensus.cc:374] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:23.507133 13026 raft_consensus.cc:374] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:23.507222 13027 raft_consensus.cc:729] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d25e4ea3264248b12ddfc599f5c718, State: Initialized, Role: FOLLOWER
I20240501 20:08:23.507309 13026 raft_consensus.cc:729] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d388ae45a29d4a15af1448cddce316af, State: Initialized, Role: FOLLOWER
I20240501 20:08:23.507534 13027 consensus_queue.cc:260] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } }
I20240501 20:08:23.507591 13026 consensus_queue.cc:260] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } }
I20240501 20:08:23.507109 13028 raft_consensus.cc:348] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } }
I20240501 20:08:23.508196 13028 raft_consensus.cc:374] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:23.508273 13028 raft_consensus.cc:729] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339dfd56e20342c1b0c3d7efb65a624b, State: Initialized, Role: FOLLOWER
I20240501 20:08:23.508473 13027 ts_tablet_manager.cc:1398] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:23.508761 13027 tablet_bootstrap.cc:492] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718: Bootstrap starting.
I20240501 20:08:23.508749 13028 consensus_queue.cc:260] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } }
I20240501 20:08:23.509311 13026 ts_tablet_manager.cc:1398] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:23.509651 13026 tablet_bootstrap.cc:492] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af: Bootstrap starting.
I20240501 20:08:23.510769 13027 tablet_bootstrap.cc:654] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:23.511376 13026 tablet_bootstrap.cc:654] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:23.516086 13028 ts_tablet_manager.cc:1398] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b: Time spent starting tablet: real 0.009s user 0.003s sys 0.001s
I20240501 20:08:23.516340 13026 tablet_bootstrap.cc:492] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af: No bootstrap required, opened a new log
I20240501 20:08:23.516479 13028 tablet_bootstrap.cc:492] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b: Bootstrap starting.
I20240501 20:08:23.516522 13026 ts_tablet_manager.cc:1367] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:08:23.517256 13026 raft_consensus.cc:348] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } }
I20240501 20:08:23.517573 13026 raft_consensus.cc:374] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:23.517670 13026 raft_consensus.cc:729] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d388ae45a29d4a15af1448cddce316af, State: Initialized, Role: FOLLOWER
I20240501 20:08:23.517932 13026 consensus_queue.cc:260] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } }
I20240501 20:08:23.518500 13028 tablet_bootstrap.cc:654] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:23.518617 13026 ts_tablet_manager.cc:1398] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:23.526842 13027 tablet_bootstrap.cc:492] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718: No bootstrap required, opened a new log
I20240501 20:08:23.526902 13028 tablet_bootstrap.cc:492] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b: No bootstrap required, opened a new log
I20240501 20:08:23.527015 13027 ts_tablet_manager.cc:1367] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.008s
I20240501 20:08:23.527047 13028 ts_tablet_manager.cc:1367] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240501 20:08:23.527663 13028 raft_consensus.cc:348] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } }
I20240501 20:08:23.527698 13027 raft_consensus.cc:348] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } }
I20240501 20:08:23.527851 13028 raft_consensus.cc:374] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:23.527863 13027 raft_consensus.cc:374] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:23.527923 13028 raft_consensus.cc:729] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339dfd56e20342c1b0c3d7efb65a624b, State: Initialized, Role: FOLLOWER
I20240501 20:08:23.527932 13027 raft_consensus.cc:729] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d25e4ea3264248b12ddfc599f5c718, State: Initialized, Role: FOLLOWER
I20240501 20:08:23.528147 13027 consensus_queue.cc:260] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } }
I20240501 20:08:23.528156 13028 consensus_queue.cc:260] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } }
I20240501 20:08:23.528672 13027 ts_tablet_manager.cc:1398] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:23.528689 13028 ts_tablet_manager.cc:1398] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:08:23.648607 13032 raft_consensus.cc:482] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:23.648924 13032 raft_consensus.cc:504] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } }
I20240501 20:08:23.650228 13032 leader_election.cc:288] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 339dfd56e20342c1b0c3d7efb65a624b (127.10.219.129:33071), d388ae45a29d4a15af1448cddce316af (127.10.219.130:36327)
I20240501 20:08:23.658535 12832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2166fbc27b124ebdb0b8ab3a6808a19d" candidate_uuid: "33d25e4ea3264248b12ddfc599f5c718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "339dfd56e20342c1b0c3d7efb65a624b" is_pre_election: true
I20240501 20:08:23.658684 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2166fbc27b124ebdb0b8ab3a6808a19d" candidate_uuid: "33d25e4ea3264248b12ddfc599f5c718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d388ae45a29d4a15af1448cddce316af" is_pre_election: true
I20240501 20:08:23.658874 12832 raft_consensus.cc:2454] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d25e4ea3264248b12ddfc599f5c718 in term 0.
I20240501 20:08:23.658962 12907 raft_consensus.cc:2454] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d25e4ea3264248b12ddfc599f5c718 in term 0.
I20240501 20:08:23.659574 12947 leader_election.cc:302] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d25e4ea3264248b12ddfc599f5c718, d388ae45a29d4a15af1448cddce316af; no voters:
I20240501 20:08:23.659973 13032 raft_consensus.cc:2781] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:23.660107 13032 raft_consensus.cc:482] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:23.660197 13032 raft_consensus.cc:3037] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:23.662626 13032 raft_consensus.cc:504] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } }
I20240501 20:08:23.663877 13032 leader_election.cc:288] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [CANDIDATE]: Term 1 election: Requested vote from peers 339dfd56e20342c1b0c3d7efb65a624b (127.10.219.129:33071), d388ae45a29d4a15af1448cddce316af (127.10.219.130:36327)
I20240501 20:08:23.664629 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2166fbc27b124ebdb0b8ab3a6808a19d" candidate_uuid: "33d25e4ea3264248b12ddfc599f5c718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d388ae45a29d4a15af1448cddce316af"
I20240501 20:08:23.664865 12907 raft_consensus.cc:3037] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:23.664958 12832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2166fbc27b124ebdb0b8ab3a6808a19d" candidate_uuid: "33d25e4ea3264248b12ddfc599f5c718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "339dfd56e20342c1b0c3d7efb65a624b"
I20240501 20:08:23.665378 12832 raft_consensus.cc:3037] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:23.668108 12907 raft_consensus.cc:2454] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d25e4ea3264248b12ddfc599f5c718 in term 1.
I20240501 20:08:23.668108 12832 raft_consensus.cc:2454] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d25e4ea3264248b12ddfc599f5c718 in term 1.
I20240501 20:08:23.668660 12947 leader_election.cc:302] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d25e4ea3264248b12ddfc599f5c718, d388ae45a29d4a15af1448cddce316af; no voters:
I20240501 20:08:23.668964 13032 raft_consensus.cc:2781] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:23.669579 13032 raft_consensus.cc:686] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 1 LEADER]: Becoming Leader. State: Replica: 33d25e4ea3264248b12ddfc599f5c718, State: Running, Role: LEADER
I20240501 20:08:23.669854 13032 consensus_queue.cc:237] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } }
I20240501 20:08:23.672996 12730 catalog_manager.cc:5484] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33d25e4ea3264248b12ddfc599f5c718 (127.10.219.131). New cstate: current_term: 1 leader_uuid: "33d25e4ea3264248b12ddfc599f5c718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:23.704165 13032 raft_consensus.cc:482] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:23.704419 13032 raft_consensus.cc:504] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } }
I20240501 20:08:23.705292 13032 leader_election.cc:288] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 339dfd56e20342c1b0c3d7efb65a624b (127.10.219.129:33071), d388ae45a29d4a15af1448cddce316af (127.10.219.130:36327)
I20240501 20:08:23.705896 12832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbbd45dd08144adbbd51cad42bad500" candidate_uuid: "33d25e4ea3264248b12ddfc599f5c718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "339dfd56e20342c1b0c3d7efb65a624b" is_pre_election: true
I20240501 20:08:23.706115 12832 raft_consensus.cc:2454] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d25e4ea3264248b12ddfc599f5c718 in term 0.
I20240501 20:08:23.705896 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbbd45dd08144adbbd51cad42bad500" candidate_uuid: "33d25e4ea3264248b12ddfc599f5c718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d388ae45a29d4a15af1448cddce316af" is_pre_election: true
I20240501 20:08:23.706513 12907 raft_consensus.cc:2454] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d25e4ea3264248b12ddfc599f5c718 in term 0.
I20240501 20:08:23.706624 12947 leader_election.cc:302] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 339dfd56e20342c1b0c3d7efb65a624b, 33d25e4ea3264248b12ddfc599f5c718; no voters:
I20240501 20:08:23.706925 13032 raft_consensus.cc:2781] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:23.707031 13032 raft_consensus.cc:482] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:23.707098 13032 raft_consensus.cc:3037] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:23.709062 13032 raft_consensus.cc:504] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } }
I20240501 20:08:23.709686 13032 leader_election.cc:288] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [CANDIDATE]: Term 1 election: Requested vote from peers 339dfd56e20342c1b0c3d7efb65a624b (127.10.219.129:33071), d388ae45a29d4a15af1448cddce316af (127.10.219.130:36327)
I20240501 20:08:23.710294 12832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbbd45dd08144adbbd51cad42bad500" candidate_uuid: "33d25e4ea3264248b12ddfc599f5c718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "339dfd56e20342c1b0c3d7efb65a624b"
I20240501 20:08:23.710386 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbbd45dd08144adbbd51cad42bad500" candidate_uuid: "33d25e4ea3264248b12ddfc599f5c718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d388ae45a29d4a15af1448cddce316af"
I20240501 20:08:23.710520 12832 raft_consensus.cc:3037] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:23.710608 12907 raft_consensus.cc:3037] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:23.712759 12832 raft_consensus.cc:2454] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d25e4ea3264248b12ddfc599f5c718 in term 1.
I20240501 20:08:23.712759 12907 raft_consensus.cc:2454] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d25e4ea3264248b12ddfc599f5c718 in term 1.
I20240501 20:08:23.713352 12947 leader_election.cc:302] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 339dfd56e20342c1b0c3d7efb65a624b, 33d25e4ea3264248b12ddfc599f5c718; no voters:
I20240501 20:08:23.713672 13032 raft_consensus.cc:2781] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:23.713831 13032 raft_consensus.cc:686] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 1 LEADER]: Becoming Leader. State: Replica: 33d25e4ea3264248b12ddfc599f5c718, State: Running, Role: LEADER
I20240501 20:08:23.714120 13032 consensus_queue.cc:237] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } }
I20240501 20:08:23.717458 12729 catalog_manager.cc:5484] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33d25e4ea3264248b12ddfc599f5c718 (127.10.219.131). New cstate: current_term: 1 leader_uuid: "33d25e4ea3264248b12ddfc599f5c718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:24.098788 13032 consensus_queue.cc:1035] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:24.104178 13032 consensus_queue.cc:1035] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:24.185736 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:24.189110 13040 consensus_queue.cc:1035] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:24.193814 12981 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2166fbc27b124ebdb0b8ab3a6808a19d"
dest_uuid: "33d25e4ea3264248b12ddfc599f5c718"
mode: GRACEFUL
new_leader_uuid: "339dfd56e20342c1b0c3d7efb65a624b"
from {username='slave'} at 127.0.0.1:49166
I20240501 20:08:24.193949 13032 consensus_queue.cc:1035] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:24.194100 12981 raft_consensus.cc:595] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 1 LEADER]: Received request to transfer leadership to TS 339dfd56e20342c1b0c3d7efb65a624b
I20240501 20:08:24.194655 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 20:08:24.194804 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:24.522825 13053 raft_consensus.cc:979] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718: : Instructing follower 339dfd56e20342c1b0c3d7efb65a624b to start an election
I20240501 20:08:24.523016 13053 raft_consensus.cc:1067] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 1 LEADER]: Signalling peer 339dfd56e20342c1b0c3d7efb65a624b to start an election
I20240501 20:08:24.523674 12832 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2166fbc27b124ebdb0b8ab3a6808a19d"
dest_uuid: "339dfd56e20342c1b0c3d7efb65a624b"
from {username='slave'} at 127.0.0.1:46952
I20240501 20:08:24.523883 12832 raft_consensus.cc:482] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:08:24.524072 12832 raft_consensus.cc:3037] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:24.526165 12832 raft_consensus.cc:504] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } }
I20240501 20:08:24.527200 12832 leader_election.cc:288] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [CANDIDATE]: Term 2 election: Requested vote from peers d388ae45a29d4a15af1448cddce316af (127.10.219.130:36327), 33d25e4ea3264248b12ddfc599f5c718 (127.10.219.131:33839)
I20240501 20:08:24.535595 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2166fbc27b124ebdb0b8ab3a6808a19d" candidate_uuid: "339dfd56e20342c1b0c3d7efb65a624b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d388ae45a29d4a15af1448cddce316af"
I20240501 20:08:24.535640 12981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2166fbc27b124ebdb0b8ab3a6808a19d" candidate_uuid: "339dfd56e20342c1b0c3d7efb65a624b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "33d25e4ea3264248b12ddfc599f5c718"
I20240501 20:08:24.535943 12907 raft_consensus.cc:3037] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:24.535946 12981 raft_consensus.cc:3032] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:08:24.536136 12981 raft_consensus.cc:729] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33d25e4ea3264248b12ddfc599f5c718, State: Running, Role: LEADER
I20240501 20:08:24.536450 12981 consensus_queue.cc:260] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } }
I20240501 20:08:24.536899 12981 raft_consensus.cc:3037] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:08:24.539173 12907 raft_consensus.cc:2454] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 339dfd56e20342c1b0c3d7efb65a624b in term 2.
I20240501 20:08:24.539615 12981 raft_consensus.cc:2454] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 339dfd56e20342c1b0c3d7efb65a624b in term 2.
I20240501 20:08:24.539778 12798 leader_election.cc:302] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 339dfd56e20342c1b0c3d7efb65a624b, d388ae45a29d4a15af1448cddce316af; no voters:
I20240501 20:08:24.540158 13043 raft_consensus.cc:2781] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:08:24.540591 13043 raft_consensus.cc:686] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 2 LEADER]: Becoming Leader. State: Replica: 339dfd56e20342c1b0c3d7efb65a624b, State: Running, Role: LEADER
I20240501 20:08:24.540915 13043 consensus_queue.cc:237] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } }
I20240501 20:08:24.546343 12730 catalog_manager.cc:5484] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b reported cstate change: term changed from 1 to 2, leader changed from 33d25e4ea3264248b12ddfc599f5c718 (127.10.219.131) to 339dfd56e20342c1b0c3d7efb65a624b (127.10.219.129). New cstate: current_term: 2 leader_uuid: "339dfd56e20342c1b0c3d7efb65a624b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:24.993777 12907 raft_consensus.cc:1261] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af [term 2 FOLLOWER]: Refusing update from remote peer 339dfd56e20342c1b0c3d7efb65a624b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:08:24.994421 13043 consensus_queue.cc:1035] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:24.999078 12981 raft_consensus.cc:1261] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 2 FOLLOWER]: Refusing update from remote peer 339dfd56e20342c1b0c3d7efb65a624b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:08:24.999780 13064 consensus_queue.cc:1035] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:26.195286 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:26.195986 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:26.196130 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:28.196684 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:28.197388 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:28.197537 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:30.198073 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:30.198892 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:30.199026 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:32.199517 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:32.200254 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:32.200454 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:34.200876 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:34.201586 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:34.201723 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:34.832814 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:34.836877 13072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:34.837632 13073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:34.838421 13075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:34.839207 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:34.839567 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:34.839646 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:34.839710 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594114839672 us; error 0 us; skew 500 ppm
I20240501 20:08:34.839891 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:34.841405 11118 webserver.cc:460] Webserver started at http://127.10.219.132:38457/ using document root <none> and password file <none>
I20240501 20:08:34.841702 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:34.841817 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:34.841985 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:34.842995 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-3-root/instance:
uuid: "728f6a73920f4b48b20e438919c96a58"
format_stamp: "Formatted at 2024-05-01 20:08:34 on dist-test-slave-n6pz"
I20240501 20:08:34.846134 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:34.848089 13080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:34.848603 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 20:08:34.848727 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-3-root
uuid: "728f6a73920f4b48b20e438919c96a58"
format_stamp: "Formatted at 2024-05-01 20:08:34 on dist-test-slave-n6pz"
I20240501 20:08:34.848883 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:34.867761 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:34.868502 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:34.869146 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:34.870205 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:34.870282 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:34.870445 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:34.870517 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:34.887877 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.132:38039
I20240501 20:08:34.887897 13142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.132:38039 every 8 connection(s)
I20240501 20:08:34.896298 13143 heartbeater.cc:347] Connected to a master server at 127.10.219.190:42981
I20240501 20:08:34.896499 13143 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:34.896953 13143 heartbeater.cc:511] Master 127.10.219.190:42981 requested a full tablet report, sending...
I20240501 20:08:34.897979 12729 ts_manager.cc:194] Registered new tserver with Master: 728f6a73920f4b48b20e438919c96a58 (127.10.219.132:38039)
I20240501 20:08:34.898926 12729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46730
I20240501 20:08:35.210296 12981 consensus_queue.cc:237] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: true } }
I20240501 20:08:35.214063 12832 raft_consensus.cc:1261] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [term 1 FOLLOWER]: Refusing update from remote peer 33d25e4ea3264248b12ddfc599f5c718: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:35.214064 12907 raft_consensus.cc:1261] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 1 FOLLOWER]: Refusing update from remote peer 33d25e4ea3264248b12ddfc599f5c718: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:35.214897 13032 consensus_queue.cc:1035] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:35.215421 13151 consensus_queue.cc:1035] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:35.219094 13032 raft_consensus.cc:2932] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 728f6a73920f4b48b20e438919c96a58 (127.10.219.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: true } } }
I20240501 20:08:35.221043 12907 raft_consensus.cc:2932] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 728f6a73920f4b48b20e438919c96a58 (127.10.219.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: true } } }
I20240501 20:08:35.222203 12832 raft_consensus.cc:2932] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 728f6a73920f4b48b20e438919c96a58 (127.10.219.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: true } } }
W20240501 20:08:35.224355 12945 consensus_peers.cc:485] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 -> Peer 728f6a73920f4b48b20e438919c96a58 (127.10.219.132:38039): Couldn't send request to peer 728f6a73920f4b48b20e438919c96a58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fbbd45dd08144adbbd51cad42bad500. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:35.226678 12729 catalog_manager.cc:5484] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 reported cstate change: config changed from index -1 to 2, NON_VOTER 728f6a73920f4b48b20e438919c96a58 (127.10.219.132) added. New cstate: current_term: 1 leader_uuid: "33d25e4ea3264248b12ddfc599f5c718" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:35.822206 13162 ts_tablet_manager.cc:902] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58: Initiating tablet copy from peer 33d25e4ea3264248b12ddfc599f5c718 (127.10.219.131:33839)
I20240501 20:08:35.824524 13162 tablet_copy_client.cc:308] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.131:33839
I20240501 20:08:35.832120 12991 tablet_copy_service.cc:140] P 33d25e4ea3264248b12ddfc599f5c718: Received BeginTabletCopySession request for tablet 6fbbd45dd08144adbbd51cad42bad500 from peer 728f6a73920f4b48b20e438919c96a58 ({username='slave'} at 127.0.0.1:36176)
I20240501 20:08:35.832346 12991 tablet_copy_service.cc:161] P 33d25e4ea3264248b12ddfc599f5c718: Beginning new tablet copy session on tablet 6fbbd45dd08144adbbd51cad42bad500 from peer 728f6a73920f4b48b20e438919c96a58 at {username='slave'} at 127.0.0.1:36176: session id = 728f6a73920f4b48b20e438919c96a58-6fbbd45dd08144adbbd51cad42bad500
I20240501 20:08:35.835937 12991 tablet_copy_source_session.cc:205] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:35.838219 13162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fbbd45dd08144adbbd51cad42bad500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.844825 13162 tablet_copy_client.cc:783] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:35.845209 13162 tablet_copy_client.cc:647] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:35.847734 13162 tablet_copy_client.cc:523] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:35.852039 13162 tablet_bootstrap.cc:492] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58: Bootstrap starting.
I20240501 20:08:35.860357 13162 tablet_bootstrap.cc:492] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:08:35.860872 13162 tablet_bootstrap.cc:492] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58: Bootstrap complete.
I20240501 20:08:35.861182 13162 ts_tablet_manager.cc:1367] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.001s
I20240501 20:08:35.861824 13162 raft_consensus.cc:348] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: true } }
I20240501 20:08:35.862100 13162 raft_consensus.cc:729] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 728f6a73920f4b48b20e438919c96a58, State: Initialized, Role: LEARNER
I20240501 20:08:35.862385 13162 consensus_queue.cc:260] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: true } }
I20240501 20:08:35.863374 13143 heartbeater.cc:503] Master 127.10.219.190:42981 was elected leader, sending a full tablet report...
I20240501 20:08:35.863416 13162 ts_tablet_manager.cc:1398] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:08:35.864212 12991 tablet_copy_service.cc:342] P 33d25e4ea3264248b12ddfc599f5c718: Request end of tablet copy session 728f6a73920f4b48b20e438919c96a58-6fbbd45dd08144adbbd51cad42bad500 received from {username='slave'} at 127.0.0.1:36176
I20240501 20:08:35.864393 12991 tablet_copy_service.cc:433] P 33d25e4ea3264248b12ddfc599f5c718: ending tablet copy session 728f6a73920f4b48b20e438919c96a58-6fbbd45dd08144adbbd51cad42bad500 on tablet 6fbbd45dd08144adbbd51cad42bad500 with peer 728f6a73920f4b48b20e438919c96a58
I20240501 20:08:36.202206 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:36.203105 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:36.203275 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:36.232681 13118 raft_consensus.cc:1203] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:08:36.857430 13173 raft_consensus.cc:1050] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718: attempting to promote NON_VOTER 728f6a73920f4b48b20e438919c96a58 to VOTER
I20240501 20:08:36.858644 13173 consensus_queue.cc:237] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false } }
I20240501 20:08:36.861546 12907 raft_consensus.cc:1261] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 1 FOLLOWER]: Refusing update from remote peer 33d25e4ea3264248b12ddfc599f5c718: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:08:36.861650 13118 raft_consensus.cc:1261] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58 [term 1 LEARNER]: Refusing update from remote peer 33d25e4ea3264248b12ddfc599f5c718: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:08:36.862401 13172 consensus_queue.cc:1035] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:08:36.865097 12832 raft_consensus.cc:1261] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [term 1 FOLLOWER]: Refusing update from remote peer 33d25e4ea3264248b12ddfc599f5c718: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:08:36.865758 13172 consensus_queue.cc:1035] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:08:36.869801 13173 raft_consensus.cc:2932] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 728f6a73920f4b48b20e438919c96a58 (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false } } }
I20240501 20:08:36.870461 12832 raft_consensus.cc:2932] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 728f6a73920f4b48b20e438919c96a58 (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false } } }
I20240501 20:08:36.873136 13172 consensus_queue.cc:1035] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:08:36.874917 13118 raft_consensus.cc:2932] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 728f6a73920f4b48b20e438919c96a58 (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false } } }
I20240501 20:08:36.876363 12730 catalog_manager.cc:5484] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b reported cstate change: config changed from index 2 to 3, 728f6a73920f4b48b20e438919c96a58 (127.10.219.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "33d25e4ea3264248b12ddfc599f5c718" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false } } }
I20240501 20:08:36.878527 12907 raft_consensus.cc:2932] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 728f6a73920f4b48b20e438919c96a58 (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "339dfd56e20342c1b0c3d7efb65a624b" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false } } }
I20240501 20:08:37.378470 12980 consensus_queue.cc:237] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false } }
I20240501 20:08:37.380527 12907 raft_consensus.cc:1261] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 1 FOLLOWER]: Refusing update from remote peer 33d25e4ea3264248b12ddfc599f5c718: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:37.380548 13118 raft_consensus.cc:1261] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58 [term 1 FOLLOWER]: Refusing update from remote peer 33d25e4ea3264248b12ddfc599f5c718: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:37.381172 13178 consensus_queue.cc:1035] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:37.381518 13173 consensus_queue.cc:1035] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:37.383890 13178 raft_consensus.cc:2932] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 339dfd56e20342c1b0c3d7efb65a624b (127.10.219.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false } } }
I20240501 20:08:37.384920 13118 raft_consensus.cc:2932] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 339dfd56e20342c1b0c3d7efb65a624b (127.10.219.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false } } }
I20240501 20:08:37.385188 12907 raft_consensus.cc:2932] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 339dfd56e20342c1b0c3d7efb65a624b (127.10.219.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false } } }
I20240501 20:08:37.387266 12717 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6fbbd45dd08144adbbd51cad42bad500 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:08:37.390377 12730 catalog_manager.cc:5484] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58 reported cstate change: config changed from index 3 to 4, VOTER 339dfd56e20342c1b0c3d7efb65a624b (127.10.219.129) evicted. New cstate: current_term: 1 leader_uuid: "33d25e4ea3264248b12ddfc599f5c718" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "33d25e4ea3264248b12ddfc599f5c718" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 33839 } } peers { permanent_uuid: "d388ae45a29d4a15af1448cddce316af" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 36327 } } peers { permanent_uuid: "728f6a73920f4b48b20e438919c96a58" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 38039 } attrs { promote: false } } }
I20240501 20:08:37.395371 12822 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbbd45dd08144adbbd51cad42bad500 with delete_type TABLET_DATA_TOMBSTONED (TS 339dfd56e20342c1b0c3d7efb65a624b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46938
I20240501 20:08:37.396327 13189 tablet_replica.cc:330] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b: stopping tablet replica
I20240501 20:08:37.396528 13189 raft_consensus.cc:2229] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:37.396747 13189 raft_consensus.cc:2258] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:37.397560 13189 ts_tablet_manager.cc:1875] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:37.402585 13189 ts_tablet_manager.cc:1888] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:08:37.402781 13189 log.cc:1196] T 6fbbd45dd08144adbbd51cad42bad500 P 339dfd56e20342c1b0c3d7efb65a624b: Deleting WAL directory at /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714594084913313-11118-0/minicluster-data/ts-0-root/wals/6fbbd45dd08144adbbd51cad42bad500
I20240501 20:08:37.403645 12717 catalog_manager.cc:4830] TS 339dfd56e20342c1b0c3d7efb65a624b (127.10.219.129:33071): tablet 6fbbd45dd08144adbbd51cad42bad500 (table test-workload [id=64c77da340c7495982e2991f98e5414f]) successfully deleted
I20240501 20:08:38.203745 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:38.204466 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:38.204627 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:40.205139 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:40.205855 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:40.205984 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:42.206509 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:42.207188 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:42.207312 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:44.207835 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:44.208560 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:44.208693 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:46.209259 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:46.210057 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:46.210202 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:48.210711 12783 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:08:48.211433 12783 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:08:48.211580 12783 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:08:48.974676 11118 tablet_server.cc:178] TabletServer@127.10.219.129:0 shutting down...
I20240501 20:08:48.988888 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:48.989199 11118 tablet_replica.cc:330] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b: stopping tablet replica
I20240501 20:08:48.989439 11118 raft_consensus.cc:2229] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:08:48.989816 11118 raft_consensus.cc:2258] T 2166fbc27b124ebdb0b8ab3a6808a19d P 339dfd56e20342c1b0c3d7efb65a624b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:49.007019 11118 tablet_server.cc:194] TabletServer@127.10.219.129:0 shutdown complete.
I20240501 20:08:49.011030 11118 tablet_server.cc:178] TabletServer@127.10.219.130:0 shutting down...
I20240501 20:08:49.022432 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:49.022938 11118 tablet_replica.cc:330] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af: stopping tablet replica
I20240501 20:08:49.023175 11118 raft_consensus.cc:2229] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:49.023404 11118 raft_consensus.cc:2258] T 2166fbc27b124ebdb0b8ab3a6808a19d P d388ae45a29d4a15af1448cddce316af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:49.024077 11118 tablet_replica.cc:330] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af: stopping tablet replica
I20240501 20:08:49.024230 11118 raft_consensus.cc:2229] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:49.024470 11118 raft_consensus.cc:2258] T 6fbbd45dd08144adbbd51cad42bad500 P d388ae45a29d4a15af1448cddce316af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:49.038924 11118 tablet_server.cc:194] TabletServer@127.10.219.130:0 shutdown complete.
I20240501 20:08:49.042507 11118 tablet_server.cc:178] TabletServer@127.10.219.131:0 shutting down...
I20240501 20:08:49.053710 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:49.054018 11118 tablet_replica.cc:330] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718: stopping tablet replica
I20240501 20:08:49.054245 11118 raft_consensus.cc:2229] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:49.054687 11118 raft_consensus.cc:2258] T 6fbbd45dd08144adbbd51cad42bad500 P 33d25e4ea3264248b12ddfc599f5c718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:49.055325 11118 tablet_replica.cc:330] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718: stopping tablet replica
I20240501 20:08:49.055480 11118 raft_consensus.cc:2229] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:49.055640 11118 raft_consensus.cc:2258] T 2166fbc27b124ebdb0b8ab3a6808a19d P 33d25e4ea3264248b12ddfc599f5c718 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:49.070415 11118 tablet_server.cc:194] TabletServer@127.10.219.131:0 shutdown complete.
I20240501 20:08:49.074393 11118 tablet_server.cc:178] TabletServer@127.10.219.132:0 shutting down...
I20240501 20:08:49.083106 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:08:49.083379 11118 tablet_replica.cc:330] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58: stopping tablet replica
I20240501 20:08:49.083585 11118 raft_consensus.cc:2229] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:49.083832 11118 raft_consensus.cc:2258] T 6fbbd45dd08144adbbd51cad42bad500 P 728f6a73920f4b48b20e438919c96a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:49.097715 11118 tablet_server.cc:194] TabletServer@127.10.219.132:0 shutdown complete.
I20240501 20:08:49.101212 11118 master.cc:537] Master@127.10.219.190:42981 shutting down...
I20240501 20:08:49.109520 11118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:08:49.109781 11118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:49.109880 11118 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9946bfe043974779b46a1c4f8ae0da9d: stopping tablet replica
I20240501 20:08:49.123696 11118 master.cc:552] Master@127.10.219.190:42981 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (27025 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240501 20:08:49.136564 11118 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.219.190:38205
I20240501 20:08:49.137149 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:49.140707 13191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:49.141564 13192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:49.142397 13194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:49.143082 11118 server_base.cc:1034] running on GCE node
I20240501 20:08:49.143329 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:08:49.143389 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:08:49.143427 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594129143421 us; error 0 us; skew 500 ppm
I20240501 20:08:49.143592 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:49.151723 11118 webserver.cc:460] Webserver started at http://127.10.219.190:38597/ using document root <none> and password file <none>
I20240501 20:08:49.152007 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:49.152079 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:49.152226 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:49.152944 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714594084913313-11118-0/minicluster-data/master-0-root/instance:
uuid: "e6d9c567b1e74695a85e1e75fb30ad27"
format_stamp: "Formatted at 2024-05-01 20:08:49 on dist-test-slave-n6pz"
I20240501 20:08:49.155853 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:08:49.157609 13199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:49.158062 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:08:49.158164 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714594084913313-11118-0/minicluster-data/master-0-root
uuid: "e6d9c567b1e74695a85e1e75fb30ad27"
format_stamp: "Formatted at 2024-05-01 20:08:49 on dist-test-slave-n6pz"
I20240501 20:08:49.158277 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714594084913313-11118-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714594084913313-11118-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714594084913313-11118-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:49.186093 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:49.186772 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:49.199381 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.190:38205
I20240501 20:08:49.199400 13250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.190:38205 every 8 connection(s)
I20240501 20:08:49.200868 13251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:49.205225 13251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27: Bootstrap starting.
I20240501 20:08:49.206558 13251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:49.208325 13251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27: No bootstrap required, opened a new log
I20240501 20:08:49.209039 13251 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d9c567b1e74695a85e1e75fb30ad27" member_type: VOTER }
I20240501 20:08:49.209184 13251 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:49.209220 13251 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d9c567b1e74695a85e1e75fb30ad27, State: Initialized, Role: FOLLOWER
I20240501 20:08:49.209435 13251 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d9c567b1e74695a85e1e75fb30ad27" member_type: VOTER }
I20240501 20:08:49.209556 13251 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:49.209614 13251 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:49.209690 13251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:49.211656 13251 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d9c567b1e74695a85e1e75fb30ad27" member_type: VOTER }
I20240501 20:08:49.211881 13251 leader_election.cc:302] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6d9c567b1e74695a85e1e75fb30ad27; no voters:
I20240501 20:08:49.212365 13251 leader_election.cc:288] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:49.212548 13254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:49.213044 13254 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [term 1 LEADER]: Becoming Leader. State: Replica: e6d9c567b1e74695a85e1e75fb30ad27, State: Running, Role: LEADER
I20240501 20:08:49.213305 13251 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:49.213284 13254 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d9c567b1e74695a85e1e75fb30ad27" member_type: VOTER }
I20240501 20:08:49.214411 13256 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6d9c567b1e74695a85e1e75fb30ad27. Latest consensus state: current_term: 1 leader_uuid: "e6d9c567b1e74695a85e1e75fb30ad27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d9c567b1e74695a85e1e75fb30ad27" member_type: VOTER } }
I20240501 20:08:49.214385 13255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6d9c567b1e74695a85e1e75fb30ad27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d9c567b1e74695a85e1e75fb30ad27" member_type: VOTER } }
I20240501 20:08:49.214684 13256 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:49.214691 13255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:49.216249 13261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:49.217685 13261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:49.221638 13261 catalog_manager.cc:1347] Generated new cluster ID: 17e6c0e0e7a6487a9173f10ca193683c
I20240501 20:08:49.221836 13261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:49.221890 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:08:49.246121 13261 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:49.247042 13261 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:49.257244 13261 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27: Generated new TSK 0
I20240501 20:08:49.257544 13261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:49.286680 11118 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.000101581s
I20240501 20:09:01.371790 11118 master.cc:537] Master@127.10.219.190:38205 shutting down...
I20240501 20:09:01.386541 11118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:01.386845 11118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:01.386945 11118 tablet_replica.cc:330] T 00000000000000000000000000000000 P e6d9c567b1e74695a85e1e75fb30ad27: stopping tablet replica
I20240501 20:09:01.401180 11118 master.cc:552] Master@127.10.219.190:38205 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12288 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240501 20:09:01.425195 11118 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.219.190:40247
I20240501 20:09:01.425819 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:01.429780 13274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:01.430559 13273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:01.431277 13276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:01.432072 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:01.432336 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:01.432399 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:01.432425 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594141432420 us; error 0 us; skew 500 ppm
I20240501 20:09:01.432592 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:01.439002 11118 webserver.cc:460] Webserver started at http://127.10.219.190:34471/ using document root <none> and password file <none>
I20240501 20:09:01.439321 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:01.439404 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:01.439553 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:01.440289 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/master-0-root/instance:
uuid: "bebe0f25fd494c04b45d5ab007dcb2ca"
format_stamp: "Formatted at 2024-05-01 20:09:01 on dist-test-slave-n6pz"
I20240501 20:09:01.443365 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 20:09:01.445390 13281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:01.445856 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:09:01.445991 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/master-0-root
uuid: "bebe0f25fd494c04b45d5ab007dcb2ca"
format_stamp: "Formatted at 2024-05-01 20:09:01 on dist-test-slave-n6pz"
I20240501 20:09:01.446128 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:01.459414 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:01.460127 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:01.473184 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.190:40247
I20240501 20:09:01.473196 13332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.190:40247 every 8 connection(s)
I20240501 20:09:01.474785 13333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:01.479190 13333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca: Bootstrap starting.
I20240501 20:09:01.480636 13333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:01.482609 13333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca: No bootstrap required, opened a new log
I20240501 20:09:01.483234 13333 raft_consensus.cc:348] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebe0f25fd494c04b45d5ab007dcb2ca" member_type: VOTER }
I20240501 20:09:01.483381 13333 raft_consensus.cc:374] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:01.483415 13333 raft_consensus.cc:729] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebe0f25fd494c04b45d5ab007dcb2ca, State: Initialized, Role: FOLLOWER
I20240501 20:09:01.483606 13333 consensus_queue.cc:260] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebe0f25fd494c04b45d5ab007dcb2ca" member_type: VOTER }
I20240501 20:09:01.483733 13333 raft_consensus.cc:388] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:01.483776 13333 raft_consensus.cc:482] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:01.483829 13333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:01.485715 13333 raft_consensus.cc:504] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebe0f25fd494c04b45d5ab007dcb2ca" member_type: VOTER }
I20240501 20:09:01.485926 13333 leader_election.cc:302] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bebe0f25fd494c04b45d5ab007dcb2ca; no voters:
I20240501 20:09:01.486388 13333 leader_election.cc:288] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:01.486639 13336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:01.487180 13336 raft_consensus.cc:686] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [term 1 LEADER]: Becoming Leader. State: Replica: bebe0f25fd494c04b45d5ab007dcb2ca, State: Running, Role: LEADER
I20240501 20:09:01.487448 13333 sys_catalog.cc:564] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:09:01.487430 13336 consensus_queue.cc:237] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebe0f25fd494c04b45d5ab007dcb2ca" member_type: VOTER }
I20240501 20:09:01.488344 13337 sys_catalog.cc:455] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bebe0f25fd494c04b45d5ab007dcb2ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebe0f25fd494c04b45d5ab007dcb2ca" member_type: VOTER } }
I20240501 20:09:01.488442 13338 sys_catalog.cc:455] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader bebe0f25fd494c04b45d5ab007dcb2ca. Latest consensus state: current_term: 1 leader_uuid: "bebe0f25fd494c04b45d5ab007dcb2ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bebe0f25fd494c04b45d5ab007dcb2ca" member_type: VOTER } }
I20240501 20:09:01.488775 13338 sys_catalog.cc:458] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:01.488655 13337 sys_catalog.cc:458] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:01.489956 13342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:09:01.492018 13342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:09:01.492920 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:09:01.495517 13342 catalog_manager.cc:1347] Generated new cluster ID: 794b900f5b7645eb95bec7ef76145900
I20240501 20:09:01.495632 13342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:09:01.503103 13342 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:09:01.504001 13342 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:09:01.510200 13342 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca: Generated new TSK 0
I20240501 20:09:01.510537 13342 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:09:01.526084 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:01.530063 13354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:01.530833 13355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:01.531601 13357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:01.532346 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:01.532625 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:01.532683 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:01.532713 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594141532710 us; error 0 us; skew 500 ppm
I20240501 20:09:01.532829 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:01.534437 11118 webserver.cc:460] Webserver started at http://127.10.219.129:45289/ using document root <none> and password file <none>
I20240501 20:09:01.534677 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:01.534744 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:01.534850 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:01.535526 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-0-root/instance:
uuid: "34550d94e01f470da3a3763e8f636074"
format_stamp: "Formatted at 2024-05-01 20:09:01 on dist-test-slave-n6pz"
I20240501 20:09:01.538460 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240501 20:09:01.540247 13362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:01.540774 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 20:09:01.540907 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-0-root
uuid: "34550d94e01f470da3a3763e8f636074"
format_stamp: "Formatted at 2024-05-01 20:09:01 on dist-test-slave-n6pz"
I20240501 20:09:01.541038 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:01.563009 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:01.563638 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:01.564306 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:01.565326 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:01.565397 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:01.565477 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:01.565513 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:01.580732 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.129:43115
I20240501 20:09:01.580749 13424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.129:43115 every 8 connection(s)
I20240501 20:09:01.582616 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:01.586741 13429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:01.587580 13430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:01.588510 13432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:01.589221 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:01.589525 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:01.589599 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:01.589650 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594141589645 us; error 0 us; skew 500 ppm
I20240501 20:09:01.589826 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:01.589916 13425 heartbeater.cc:347] Connected to a master server at 127.10.219.190:40247
I20240501 20:09:01.590051 13425 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:01.590517 13425 heartbeater.cc:511] Master 127.10.219.190:40247 requested a full tablet report, sending...
I20240501 20:09:01.591424 11118 webserver.cc:460] Webserver started at http://127.10.219.130:39969/ using document root <none> and password file <none>
I20240501 20:09:01.591578 13298 ts_manager.cc:194] Registered new tserver with Master: 34550d94e01f470da3a3763e8f636074 (127.10.219.129:43115)
I20240501 20:09:01.591668 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:01.591753 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:01.591877 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:01.592523 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-1-root/instance:
uuid: "74d0d007ace04cb581f716400a842991"
format_stamp: "Formatted at 2024-05-01 20:09:01 on dist-test-slave-n6pz"
I20240501 20:09:01.592702 13298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54442
I20240501 20:09:01.595758 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:09:01.597507 13437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:01.597877 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:09:01.597988 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-1-root
uuid: "74d0d007ace04cb581f716400a842991"
format_stamp: "Formatted at 2024-05-01 20:09:01 on dist-test-slave-n6pz"
I20240501 20:09:01.598100 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:01.608642 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:01.609253 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:01.609840 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:01.610780 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:01.610842 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:01.610924 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:01.610960 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:01.624301 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.130:40539
I20240501 20:09:01.624336 13499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.130:40539 every 8 connection(s)
I20240501 20:09:01.625975 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:01.630267 13503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:01.631047 13504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:01.631831 13506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:01.632534 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:01.632849 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:01.632910 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:01.632943 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594141632939 us; error 0 us; skew 500 ppm
I20240501 20:09:01.633083 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:01.633136 13500 heartbeater.cc:347] Connected to a master server at 127.10.219.190:40247
I20240501 20:09:01.633272 13500 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:01.633750 13500 heartbeater.cc:511] Master 127.10.219.190:40247 requested a full tablet report, sending...
I20240501 20:09:01.634737 11118 webserver.cc:460] Webserver started at http://127.10.219.131:37961/ using document root <none> and password file <none>
I20240501 20:09:01.634986 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:01.635079 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:01.635052 13298 ts_manager.cc:194] Registered new tserver with Master: 74d0d007ace04cb581f716400a842991 (127.10.219.130:40539)
I20240501 20:09:01.635246 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:01.635915 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-2-root/instance:
uuid: "272134aca98e479e9808249dae0b2b2f"
format_stamp: "Formatted at 2024-05-01 20:09:01 on dist-test-slave-n6pz"
I20240501 20:09:01.636224 13298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54444
I20240501 20:09:01.638831 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 20:09:01.640561 13511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:01.640957 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:09:01.641060 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-2-root
uuid: "272134aca98e479e9808249dae0b2b2f"
format_stamp: "Formatted at 2024-05-01 20:09:01 on dist-test-slave-n6pz"
I20240501 20:09:01.641208 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714594084913313-11118-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:01.663627 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:01.664247 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:01.664867 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:01.665817 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:01.665877 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:01.665952 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:01.665993 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:01.681139 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.131:37285
I20240501 20:09:01.681159 13573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.131:37285 every 8 connection(s)
I20240501 20:09:01.688498 13574 heartbeater.cc:347] Connected to a master server at 127.10.219.190:40247
I20240501 20:09:01.688663 13574 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:01.689061 13574 heartbeater.cc:511] Master 127.10.219.190:40247 requested a full tablet report, sending...
I20240501 20:09:01.689990 13298 ts_manager.cc:194] Registered new tserver with Master: 272134aca98e479e9808249dae0b2b2f (127.10.219.131:37285)
I20240501 20:09:01.690122 11118 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007956248s
I20240501 20:09:01.690897 13298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54450
I20240501 20:09:02.594378 13425 heartbeater.cc:503] Master 127.10.219.190:40247 was elected leader, sending a full tablet report...
I20240501 20:09:02.637701 13500 heartbeater.cc:503] Master 127.10.219.190:40247 was elected leader, sending a full tablet report...
I20240501 20:09:02.692432 13574 heartbeater.cc:503] Master 127.10.219.190:40247 was elected leader, sending a full tablet report...
I20240501 20:09:03.492846 13351 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:05.493278 13351 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:07.493837 13351 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:09.494381 13351 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:11.494904 13351 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:13.495543 13351 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:13.775650 11118 tablet_server.cc:178] TabletServer@127.10.219.129:0 shutting down...
I20240501 20:09:13.790081 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:09:13.804558 11118 tablet_server.cc:194] TabletServer@127.10.219.129:0 shutdown complete.
I20240501 20:09:13.807902 11118 tablet_server.cc:178] TabletServer@127.10.219.130:0 shutting down...
I20240501 20:09:13.819300 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:09:13.833246 11118 tablet_server.cc:194] TabletServer@127.10.219.130:0 shutdown complete.
I20240501 20:09:13.837808 11118 tablet_server.cc:178] TabletServer@127.10.219.131:0 shutting down...
I20240501 20:09:13.847491 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:09:13.861003 11118 tablet_server.cc:194] TabletServer@127.10.219.131:0 shutdown complete.
I20240501 20:09:13.864351 11118 master.cc:537] Master@127.10.219.190:40247 shutting down...
I20240501 20:09:13.872915 11118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:13.873229 11118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:13.873351 11118 tablet_replica.cc:330] T 00000000000000000000000000000000 P bebe0f25fd494c04b45d5ab007dcb2ca: stopping tablet replica
I20240501 20:09:13.888288 11118 master.cc:552] Master@127.10.219.190:40247 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12474 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240501 20:09:13.899947 11118 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.219.190:39005
I20240501 20:09:13.900573 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:13.904196 13578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:13.905016 13579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:13.905771 13581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:13.906744 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:13.917922 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:13.918053 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:13.918081 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594153918077 us; error 0 us; skew 500 ppm
I20240501 20:09:13.918229 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:13.920006 11118 webserver.cc:460] Webserver started at http://127.10.219.190:34037/ using document root <none> and password file <none>
I20240501 20:09:13.920258 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:13.920320 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:13.920437 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:13.921145 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/master-0-root/instance:
uuid: "77ad3e8ef82440dd9ea5e91fd09688e1"
format_stamp: "Formatted at 2024-05-01 20:09:13 on dist-test-slave-n6pz"
I20240501 20:09:13.924165 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:09:13.926151 13586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:13.926648 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 20:09:13.926782 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/master-0-root
uuid: "77ad3e8ef82440dd9ea5e91fd09688e1"
format_stamp: "Formatted at 2024-05-01 20:09:13 on dist-test-slave-n6pz"
I20240501 20:09:13.926910 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:13.945437 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:13.946105 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:13.960635 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.190:39005
I20240501 20:09:13.960664 13637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.190:39005 every 8 connection(s)
I20240501 20:09:13.962227 13638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:13.967645 13638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1: Bootstrap starting.
I20240501 20:09:13.968959 13638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:13.970774 13638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1: No bootstrap required, opened a new log
I20240501 20:09:13.971446 13638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ad3e8ef82440dd9ea5e91fd09688e1" member_type: VOTER }
I20240501 20:09:13.971596 13638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:13.971630 13638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ad3e8ef82440dd9ea5e91fd09688e1, State: Initialized, Role: FOLLOWER
I20240501 20:09:13.971828 13638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ad3e8ef82440dd9ea5e91fd09688e1" member_type: VOTER }
I20240501 20:09:13.971940 13638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:13.971998 13638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:13.972074 13638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:13.973997 13638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ad3e8ef82440dd9ea5e91fd09688e1" member_type: VOTER }
I20240501 20:09:13.974213 13638 leader_election.cc:302] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77ad3e8ef82440dd9ea5e91fd09688e1; no voters:
I20240501 20:09:13.974738 13638 leader_election.cc:288] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:13.974911 13641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:13.975432 13641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [term 1 LEADER]: Becoming Leader. State: Replica: 77ad3e8ef82440dd9ea5e91fd09688e1, State: Running, Role: LEADER
I20240501 20:09:13.975687 13638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:09:13.975703 13641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ad3e8ef82440dd9ea5e91fd09688e1" member_type: VOTER }
I20240501 20:09:13.976759 13643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77ad3e8ef82440dd9ea5e91fd09688e1. Latest consensus state: current_term: 1 leader_uuid: "77ad3e8ef82440dd9ea5e91fd09688e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ad3e8ef82440dd9ea5e91fd09688e1" member_type: VOTER } }
I20240501 20:09:13.976955 13643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:13.977690 13642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77ad3e8ef82440dd9ea5e91fd09688e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ad3e8ef82440dd9ea5e91fd09688e1" member_type: VOTER } }
I20240501 20:09:13.977999 13642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:13.978888 13648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:09:13.980440 13648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:09:13.981969 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:09:13.983841 13648 catalog_manager.cc:1347] Generated new cluster ID: 070208a6a53648138a1ca9c43a58384b
I20240501 20:09:13.983928 13648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:09:13.998569 13648 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:09:13.999456 13648 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:09:14.010397 13648 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1: Generated new TSK 0
I20240501 20:09:14.010704 13648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:09:14.015033 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:14.018854 13659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:14.019742 13660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:14.020535 13662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:14.021400 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:14.021740 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:14.021804 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:14.021844 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594154021840 us; error 0 us; skew 500 ppm
I20240501 20:09:14.022008 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:14.023487 11118 webserver.cc:460] Webserver started at http://127.10.219.129:38727/ using document root <none> and password file <none>
I20240501 20:09:14.023754 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:14.023825 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:14.023967 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:14.024708 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-0-root/instance:
uuid: "b875325535d94b1a826a5a7b25b1d0f6"
format_stamp: "Formatted at 2024-05-01 20:09:14 on dist-test-slave-n6pz"
I20240501 20:09:14.027906 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:09:14.029855 13667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.030230 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:09:14.030390 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-0-root
uuid: "b875325535d94b1a826a5a7b25b1d0f6"
format_stamp: "Formatted at 2024-05-01 20:09:14 on dist-test-slave-n6pz"
I20240501 20:09:14.030521 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:14.050460 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:14.051115 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:14.051760 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:14.052759 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:14.052824 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.052901 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:14.052947 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.067608 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.129:41209
I20240501 20:09:14.067647 13729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.129:41209 every 8 connection(s)
I20240501 20:09:14.069461 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:14.073959 13734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:14.074806 13735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:14.080404 13730 heartbeater.cc:347] Connected to a master server at 127.10.219.190:39005
I20240501 20:09:14.080600 13730 heartbeater.cc:464] Registering TS with master...
W20240501 20:09:14.080857 13737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:14.081056 13730 heartbeater.cc:511] Master 127.10.219.190:39005 requested a full tablet report, sending...
I20240501 20:09:14.081750 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:14.082116 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:14.082198 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:14.082242 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594154082238 us; error 0 us; skew 500 ppm
I20240501 20:09:14.082348 13603 ts_manager.cc:194] Registered new tserver with Master: b875325535d94b1a826a5a7b25b1d0f6 (127.10.219.129:41209)
I20240501 20:09:14.082445 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:14.084173 13603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49398
I20240501 20:09:14.084241 11118 webserver.cc:460] Webserver started at http://127.10.219.130:35561/ using document root <none> and password file <none>
I20240501 20:09:14.084512 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:14.084609 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:14.084754 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:14.085475 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-1-root/instance:
uuid: "f461d5df16d449efbb65c659c087a498"
format_stamp: "Formatted at 2024-05-01 20:09:14 on dist-test-slave-n6pz"
I20240501 20:09:14.088440 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:09:14.090579 13742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.091038 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:09:14.091168 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-1-root
uuid: "f461d5df16d449efbb65c659c087a498"
format_stamp: "Formatted at 2024-05-01 20:09:14 on dist-test-slave-n6pz"
I20240501 20:09:14.091302 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:14.106201 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:14.107041 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:14.107740 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:14.108700 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:14.108768 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.108846 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:14.108886 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.125036 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.130:33123
I20240501 20:09:14.125056 13804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.130:33123 every 8 connection(s)
I20240501 20:09:14.126915 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:14.131275 13809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:14.132074 13808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:14.132814 13811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:14.133651 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:14.134002 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:14.134084 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:14.134110 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594154134107 us; error 0 us; skew 500 ppm
I20240501 20:09:14.134234 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:14.134270 13805 heartbeater.cc:347] Connected to a master server at 127.10.219.190:39005
I20240501 20:09:14.134454 13805 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:14.134821 13805 heartbeater.cc:511] Master 127.10.219.190:39005 requested a full tablet report, sending...
I20240501 20:09:14.135883 11118 webserver.cc:460] Webserver started at http://127.10.219.131:41735/ using document root <none> and password file <none>
I20240501 20:09:14.135921 13603 ts_manager.cc:194] Registered new tserver with Master: f461d5df16d449efbb65c659c087a498 (127.10.219.130:33123)
I20240501 20:09:14.136137 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:14.136241 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:14.136384 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:14.136998 13603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49410
I20240501 20:09:14.137271 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-2-root/instance:
uuid: "93a22bab01f149928b2c83e95940ca86"
format_stamp: "Formatted at 2024-05-01 20:09:14 on dist-test-slave-n6pz"
I20240501 20:09:14.140550 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 20:09:14.142644 13816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.143049 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:09:14.143173 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-2-root
uuid: "93a22bab01f149928b2c83e95940ca86"
format_stamp: "Formatted at 2024-05-01 20:09:14 on dist-test-slave-n6pz"
I20240501 20:09:14.143342 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:14.153143 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:14.153823 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:14.154538 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:14.155467 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:14.155555 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.155637 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:14.155675 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.171262 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.131:32993
I20240501 20:09:14.171293 13878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.131:32993 every 8 connection(s)
I20240501 20:09:14.173123 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:14.178818 13882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:14.179512 13883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:14.180498 11118 server_base.cc:1034] running on GCE node
W20240501 20:09:14.181084 13885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:14.181756 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:14.181864 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:14.181897 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594154181892 us; error 0 us; skew 500 ppm
I20240501 20:09:14.182062 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:14.183914 13879 heartbeater.cc:347] Connected to a master server at 127.10.219.190:39005
I20240501 20:09:14.184100 13879 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:14.184386 11118 webserver.cc:460] Webserver started at http://127.10.219.132:38727/ using document root <none> and password file <none>
I20240501 20:09:14.184624 13879 heartbeater.cc:511] Master 127.10.219.190:39005 requested a full tablet report, sending...
I20240501 20:09:14.184681 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:14.184757 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:14.184908 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:14.185714 13603 ts_manager.cc:194] Registered new tserver with Master: 93a22bab01f149928b2c83e95940ca86 (127.10.219.131:32993)
I20240501 20:09:14.185711 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-3-root/instance:
uuid: "d7b42c7ac4f44afd98d93b495535f9d9"
format_stamp: "Formatted at 2024-05-01 20:09:14 on dist-test-slave-n6pz"
I20240501 20:09:14.186657 13603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49426
I20240501 20:09:14.189113 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240501 20:09:14.191141 13890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.191587 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:09:14.191697 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-3-root
uuid: "d7b42c7ac4f44afd98d93b495535f9d9"
format_stamp: "Formatted at 2024-05-01 20:09:14 on dist-test-slave-n6pz"
I20240501 20:09:14.191871 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714594084913313-11118-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:14.218775 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:14.219465 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:14.220098 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:14.221060 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:14.221120 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.221199 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:14.221248 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.236884 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.132:33043
I20240501 20:09:14.236904 13952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.132:33043 every 8 connection(s)
I20240501 20:09:14.244650 13953 heartbeater.cc:347] Connected to a master server at 127.10.219.190:39005
I20240501 20:09:14.244832 13953 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:14.245209 13953 heartbeater.cc:511] Master 127.10.219.190:39005 requested a full tablet report, sending...
I20240501 20:09:14.246217 13603 ts_manager.cc:194] Registered new tserver with Master: d7b42c7ac4f44afd98d93b495535f9d9 (127.10.219.132:33043)
I20240501 20:09:14.246991 11118 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.009082181s
I20240501 20:09:14.247097 13603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49432
I20240501 20:09:15.085906 13730 heartbeater.cc:503] Master 127.10.219.190:39005 was elected leader, sending a full tablet report...
I20240501 20:09:15.138813 13805 heartbeater.cc:503] Master 127.10.219.190:39005 was elected leader, sending a full tablet report...
I20240501 20:09:15.188380 13879 heartbeater.cc:503] Master 127.10.219.190:39005 was elected leader, sending a full tablet report...
I20240501 20:09:15.248677 13953 heartbeater.cc:503] Master 127.10.219.190:39005 was elected leader, sending a full tablet report...
I20240501 20:09:15.284372 11118 test_util.cc:268] Using random seed: 261048205
I20240501 20:09:15.297463 13603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49442:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 20:09:15.323637 13695 tablet_service.cc:1467] Processing CreateTablet for tablet e9b8511524f045e1ab620237a6a67ef1 (DEFAULT_TABLE table=test-workload [id=0415407d99f444c4bea804d8f8faf39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.324259 13695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b8511524f045e1ab620237a6a67ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.327602 13918 tablet_service.cc:1467] Processing CreateTablet for tablet e9b8511524f045e1ab620237a6a67ef1 (DEFAULT_TABLE table=test-workload [id=0415407d99f444c4bea804d8f8faf39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.328109 13918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b8511524f045e1ab620237a6a67ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.328490 13844 tablet_service.cc:1467] Processing CreateTablet for tablet e9b8511524f045e1ab620237a6a67ef1 (DEFAULT_TABLE table=test-workload [id=0415407d99f444c4bea804d8f8faf39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:15.328938 13844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b8511524f045e1ab620237a6a67ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.334978 13974 tablet_bootstrap.cc:492] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6: Bootstrap starting.
I20240501 20:09:15.337064 13974 tablet_bootstrap.cc:654] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.338940 13976 tablet_bootstrap.cc:492] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86: Bootstrap starting.
I20240501 20:09:15.342043 13976 tablet_bootstrap.cc:654] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.343104 13975 tablet_bootstrap.cc:492] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9: Bootstrap starting.
I20240501 20:09:15.345594 13974 tablet_bootstrap.cc:492] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6: No bootstrap required, opened a new log
I20240501 20:09:15.345726 13976 tablet_bootstrap.cc:492] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86: No bootstrap required, opened a new log
I20240501 20:09:15.345779 13974 ts_tablet_manager.cc:1367] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240501 20:09:15.345880 13976 ts_tablet_manager.cc:1367] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:09:15.346217 13975 tablet_bootstrap.cc:654] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:15.346681 13974 raft_consensus.cc:348] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b875325535d94b1a826a5a7b25b1d0f6" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41209 } } peers { permanent_uuid: "d7b42c7ac4f44afd98d93b495535f9d9" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 33043 } } peers { permanent_uuid: "93a22bab01f149928b2c83e95940ca86" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 32993 } }
I20240501 20:09:15.346887 13974 raft_consensus.cc:374] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.346819 13976 raft_consensus.cc:348] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b875325535d94b1a826a5a7b25b1d0f6" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41209 } } peers { permanent_uuid: "d7b42c7ac4f44afd98d93b495535f9d9" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 33043 } } peers { permanent_uuid: "93a22bab01f149928b2c83e95940ca86" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 32993 } }
I20240501 20:09:15.346977 13974 raft_consensus.cc:729] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b875325535d94b1a826a5a7b25b1d0f6, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.347110 13976 raft_consensus.cc:374] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.347184 13976 raft_consensus.cc:729] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a22bab01f149928b2c83e95940ca86, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.347322 13974 consensus_queue.cc:260] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b875325535d94b1a826a5a7b25b1d0f6" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41209 } } peers { permanent_uuid: "d7b42c7ac4f44afd98d93b495535f9d9" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 33043 } } peers { permanent_uuid: "93a22bab01f149928b2c83e95940ca86" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 32993 } }
I20240501 20:09:15.347419 13976 consensus_queue.cc:260] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b875325535d94b1a826a5a7b25b1d0f6" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41209 } } peers { permanent_uuid: "d7b42c7ac4f44afd98d93b495535f9d9" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 33043 } } peers { permanent_uuid: "93a22bab01f149928b2c83e95940ca86" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 32993 } }
I20240501 20:09:15.348289 13974 ts_tablet_manager.cc:1398] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:15.349085 13975 tablet_bootstrap.cc:492] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9: No bootstrap required, opened a new log
I20240501 20:09:15.349288 13975 ts_tablet_manager.cc:1367] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:09:15.350531 13975 raft_consensus.cc:348] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b875325535d94b1a826a5a7b25b1d0f6" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41209 } } peers { permanent_uuid: "d7b42c7ac4f44afd98d93b495535f9d9" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 33043 } } peers { permanent_uuid: "93a22bab01f149928b2c83e95940ca86" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 32993 } }
I20240501 20:09:15.350793 13975 raft_consensus.cc:374] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:15.350849 13975 raft_consensus.cc:729] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b42c7ac4f44afd98d93b495535f9d9, State: Initialized, Role: FOLLOWER
I20240501 20:09:15.351073 13975 consensus_queue.cc:260] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b875325535d94b1a826a5a7b25b1d0f6" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41209 } } peers { permanent_uuid: "d7b42c7ac4f44afd98d93b495535f9d9" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 33043 } } peers { permanent_uuid: "93a22bab01f149928b2c83e95940ca86" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 32993 } }
I20240501 20:09:15.354986 13975 ts_tablet_manager.cc:1398] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 20:09:15.355569 13976 ts_tablet_manager.cc:1398] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
W20240501 20:09:15.423733 13880 tablet.cc:2369] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:15.470110 13980 raft_consensus.cc:482] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:15.470391 13980 raft_consensus.cc:504] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b875325535d94b1a826a5a7b25b1d0f6" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41209 } } peers { permanent_uuid: "d7b42c7ac4f44afd98d93b495535f9d9" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 33043 } } peers { permanent_uuid: "93a22bab01f149928b2c83e95940ca86" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 32993 } }
I20240501 20:09:15.471282 13980 leader_election.cc:288] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7b42c7ac4f44afd98d93b495535f9d9 (127.10.219.132:33043), 93a22bab01f149928b2c83e95940ca86 (127.10.219.131:32993)
I20240501 20:09:15.479071 13928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b8511524f045e1ab620237a6a67ef1" candidate_uuid: "b875325535d94b1a826a5a7b25b1d0f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b42c7ac4f44afd98d93b495535f9d9" is_pre_election: true
I20240501 20:09:15.479071 13854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b8511524f045e1ab620237a6a67ef1" candidate_uuid: "b875325535d94b1a826a5a7b25b1d0f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a22bab01f149928b2c83e95940ca86" is_pre_election: true
I20240501 20:09:15.479492 13928 raft_consensus.cc:2454] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b875325535d94b1a826a5a7b25b1d0f6 in term 0.
I20240501 20:09:15.479523 13854 raft_consensus.cc:2454] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b875325535d94b1a826a5a7b25b1d0f6 in term 0.
I20240501 20:09:15.479982 13668 leader_election.cc:302] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b875325535d94b1a826a5a7b25b1d0f6, d7b42c7ac4f44afd98d93b495535f9d9; no voters:
I20240501 20:09:15.480265 13980 raft_consensus.cc:2781] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:15.480367 13980 raft_consensus.cc:482] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:15.480450 13980 raft_consensus.cc:3037] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.482540 13980 raft_consensus.cc:504] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b875325535d94b1a826a5a7b25b1d0f6" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41209 } } peers { permanent_uuid: "d7b42c7ac4f44afd98d93b495535f9d9" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 33043 } } peers { permanent_uuid: "93a22bab01f149928b2c83e95940ca86" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 32993 } }
I20240501 20:09:15.483161 13980 leader_election.cc:288] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [CANDIDATE]: Term 1 election: Requested vote from peers d7b42c7ac4f44afd98d93b495535f9d9 (127.10.219.132:33043), 93a22bab01f149928b2c83e95940ca86 (127.10.219.131:32993)
I20240501 20:09:15.483717 13928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b8511524f045e1ab620237a6a67ef1" candidate_uuid: "b875325535d94b1a826a5a7b25b1d0f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b42c7ac4f44afd98d93b495535f9d9"
I20240501 20:09:15.483901 13928 raft_consensus.cc:3037] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.483857 13854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b8511524f045e1ab620237a6a67ef1" candidate_uuid: "b875325535d94b1a826a5a7b25b1d0f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a22bab01f149928b2c83e95940ca86"
I20240501 20:09:15.484019 13854 raft_consensus.cc:3037] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:15.486267 13928 raft_consensus.cc:2454] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b875325535d94b1a826a5a7b25b1d0f6 in term 1.
I20240501 20:09:15.486606 13854 raft_consensus.cc:2454] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b875325535d94b1a826a5a7b25b1d0f6 in term 1.
I20240501 20:09:15.486786 13668 leader_election.cc:302] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b875325535d94b1a826a5a7b25b1d0f6, d7b42c7ac4f44afd98d93b495535f9d9; no voters:
I20240501 20:09:15.487069 13980 raft_consensus.cc:2781] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:15.487540 13980 raft_consensus.cc:686] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 1 LEADER]: Becoming Leader. State: Replica: b875325535d94b1a826a5a7b25b1d0f6, State: Running, Role: LEADER
I20240501 20:09:15.487803 13980 consensus_queue.cc:237] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b875325535d94b1a826a5a7b25b1d0f6" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41209 } } peers { permanent_uuid: "d7b42c7ac4f44afd98d93b495535f9d9" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 33043 } } peers { permanent_uuid: "93a22bab01f149928b2c83e95940ca86" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 32993 } }
I20240501 20:09:15.490567 13603 catalog_manager.cc:5484] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b875325535d94b1a826a5a7b25b1d0f6 (127.10.219.129). New cstate: current_term: 1 leader_uuid: "b875325535d94b1a826a5a7b25b1d0f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b875325535d94b1a826a5a7b25b1d0f6" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b42c7ac4f44afd98d93b495535f9d9" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 33043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93a22bab01f149928b2c83e95940ca86" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 32993 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:15.929906 13980 consensus_queue.cc:1035] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b42c7ac4f44afd98d93b495535f9d9" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:15.936117 13990 consensus_queue.cc:1035] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93a22bab01f149928b2c83e95940ca86" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:15.981920 13656 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:15.982508 13656 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:15.982647 13656 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:17.983093 13656 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:17.983594 13656 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:17.983700 13656 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:19.984171 13656 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:19.984737 13656 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:19.984849 13656 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:21.985248 13656 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:21.985781 13656 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:21.985911 13656 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:23.986375 13656 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:23.986985 13656 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:23.987133 13656 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:25.987658 13656 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:25.988204 13656 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:25.988334 13656 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:26.569206 11118 tablet_server.cc:178] TabletServer@127.10.219.129:0 shutting down...
I20240501 20:09:26.582549 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:09:26.582921 11118 tablet_replica.cc:330] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6: stopping tablet replica
I20240501 20:09:26.583148 11118 raft_consensus.cc:2229] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:26.583606 11118 raft_consensus.cc:2258] T e9b8511524f045e1ab620237a6a67ef1 P b875325535d94b1a826a5a7b25b1d0f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:26.599256 11118 tablet_server.cc:194] TabletServer@127.10.219.129:0 shutdown complete.
I20240501 20:09:26.614296 11118 tablet_server.cc:178] TabletServer@127.10.219.130:0 shutting down...
I20240501 20:09:26.623346 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:09:26.636400 11118 tablet_server.cc:194] TabletServer@127.10.219.130:0 shutdown complete.
I20240501 20:09:26.639451 11118 tablet_server.cc:178] TabletServer@127.10.219.131:0 shutting down...
I20240501 20:09:26.648841 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:09:26.649108 11118 tablet_replica.cc:330] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86: stopping tablet replica
I20240501 20:09:26.649284 11118 raft_consensus.cc:2229] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:26.649479 11118 raft_consensus.cc:2258] T e9b8511524f045e1ab620237a6a67ef1 P 93a22bab01f149928b2c83e95940ca86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:26.663172 11118 tablet_server.cc:194] TabletServer@127.10.219.131:0 shutdown complete.
I20240501 20:09:26.666705 11118 tablet_server.cc:178] TabletServer@127.10.219.132:0 shutting down...
I20240501 20:09:26.677373 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:09:26.677678 11118 tablet_replica.cc:330] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9: stopping tablet replica
I20240501 20:09:26.677853 11118 raft_consensus.cc:2229] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:26.678040 11118 raft_consensus.cc:2258] T e9b8511524f045e1ab620237a6a67ef1 P d7b42c7ac4f44afd98d93b495535f9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:26.692059 11118 tablet_server.cc:194] TabletServer@127.10.219.132:0 shutdown complete.
I20240501 20:09:26.695940 11118 master.cc:537] Master@127.10.219.190:39005 shutting down...
I20240501 20:09:26.704459 11118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:26.704762 11118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:26.704874 11118 tablet_replica.cc:330] T 00000000000000000000000000000000 P 77ad3e8ef82440dd9ea5e91fd09688e1: stopping tablet replica
I20240501 20:09:26.719059 11118 master.cc:552] Master@127.10.219.190:39005 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12832 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240501 20:09:26.732609 11118 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.219.190:42403
I20240501 20:09:26.733218 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:26.737097 14000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:26.738118 14001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:26.738927 14003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:26.739730 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:26.739987 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:26.740049 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:26.740088 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594166740084 us; error 0 us; skew 500 ppm
I20240501 20:09:26.740252 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:26.741771 11118 webserver.cc:460] Webserver started at http://127.10.219.190:41139/ using document root <none> and password file <none>
I20240501 20:09:26.742051 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:26.742121 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:26.742256 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:26.743050 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/master-0-root/instance:
uuid: "b37000050a6f46ff97e7b0f4088acb3a"
format_stamp: "Formatted at 2024-05-01 20:09:26 on dist-test-slave-n6pz"
I20240501 20:09:26.746088 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:26.748325 14008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:26.748932 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 20:09:26.749083 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/master-0-root
uuid: "b37000050a6f46ff97e7b0f4088acb3a"
format_stamp: "Formatted at 2024-05-01 20:09:26 on dist-test-slave-n6pz"
I20240501 20:09:26.749235 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:26.782397 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:26.783139 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:26.798828 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.190:42403
I20240501 20:09:26.798828 14059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.190:42403 every 8 connection(s)
I20240501 20:09:26.800483 14060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.805063 14060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a: Bootstrap starting.
I20240501 20:09:26.806538 14060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:26.808473 14060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a: No bootstrap required, opened a new log
I20240501 20:09:26.809161 14060 raft_consensus.cc:348] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37000050a6f46ff97e7b0f4088acb3a" member_type: VOTER }
I20240501 20:09:26.809340 14060 raft_consensus.cc:374] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:26.809376 14060 raft_consensus.cc:729] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37000050a6f46ff97e7b0f4088acb3a, State: Initialized, Role: FOLLOWER
I20240501 20:09:26.809715 14060 consensus_queue.cc:260] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37000050a6f46ff97e7b0f4088acb3a" member_type: VOTER }
I20240501 20:09:26.810016 14060 raft_consensus.cc:388] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:26.810084 14060 raft_consensus.cc:482] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:26.810158 14060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:26.813128 14060 raft_consensus.cc:504] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37000050a6f46ff97e7b0f4088acb3a" member_type: VOTER }
I20240501 20:09:26.813426 14060 leader_election.cc:302] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b37000050a6f46ff97e7b0f4088acb3a; no voters:
I20240501 20:09:26.813875 14060 leader_election.cc:288] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:26.814085 14063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:26.814720 14063 raft_consensus.cc:686] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [term 1 LEADER]: Becoming Leader. State: Replica: b37000050a6f46ff97e7b0f4088acb3a, State: Running, Role: LEADER
I20240501 20:09:26.815045 14060 sys_catalog.cc:564] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:09:26.815011 14063 consensus_queue.cc:237] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37000050a6f46ff97e7b0f4088acb3a" member_type: VOTER }
I20240501 20:09:26.816090 14065 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b37000050a6f46ff97e7b0f4088acb3a. Latest consensus state: current_term: 1 leader_uuid: "b37000050a6f46ff97e7b0f4088acb3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37000050a6f46ff97e7b0f4088acb3a" member_type: VOTER } }
I20240501 20:09:26.816066 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b37000050a6f46ff97e7b0f4088acb3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37000050a6f46ff97e7b0f4088acb3a" member_type: VOTER } }
I20240501 20:09:26.816414 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:26.816407 14065 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:26.817863 14068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:09:26.820240 14068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:09:26.820844 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:09:26.824162 14068 catalog_manager.cc:1347] Generated new cluster ID: 835ad4d84b064ac696a7eb27889d9356
I20240501 20:09:26.824326 14068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:09:26.836853 14068 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:09:26.837720 14068 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:09:26.845350 14068 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a: Generated new TSK 0
I20240501 20:09:26.845700 14068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:09:26.853997 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:26.858140 14081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:26.859319 14082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:26.860532 14084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:26.861891 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:26.862196 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:26.862257 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:26.862284 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594166862280 us; error 0 us; skew 500 ppm
I20240501 20:09:26.862449 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:26.864003 11118 webserver.cc:460] Webserver started at http://127.10.219.129:44695/ using document root <none> and password file <none>
I20240501 20:09:26.864261 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:26.864334 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:26.864486 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:26.865218 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-0-root/instance:
uuid: "a11573fa0a8f4ba79f8534c441e1c15f"
format_stamp: "Formatted at 2024-05-01 20:09:26 on dist-test-slave-n6pz"
I20240501 20:09:26.868232 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:09:26.870174 14089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:26.870599 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 20:09:26.870718 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-0-root
uuid: "a11573fa0a8f4ba79f8534c441e1c15f"
format_stamp: "Formatted at 2024-05-01 20:09:26 on dist-test-slave-n6pz"
I20240501 20:09:26.870841 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:26.883463 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:26.884107 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:26.884717 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:26.885696 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:26.885756 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:26.885843 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:26.885879 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:26.900166 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.129:41995
I20240501 20:09:26.900174 14151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.129:41995 every 8 connection(s)
I20240501 20:09:26.902009 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:26.906188 14156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:26.907146 14157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:26.909130 14152 heartbeater.cc:347] Connected to a master server at 127.10.219.190:42403
I20240501 20:09:26.909271 14152 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:26.909649 14152 heartbeater.cc:511] Master 127.10.219.190:42403 requested a full tablet report, sending...
W20240501 20:09:26.910151 14159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:26.910719 14025 ts_manager.cc:194] Registered new tserver with Master: a11573fa0a8f4ba79f8534c441e1c15f (127.10.219.129:41995)
I20240501 20:09:26.910868 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:26.911149 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:26.911209 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:26.911252 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594166911247 us; error 0 us; skew 500 ppm
I20240501 20:09:26.911378 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:26.912221 14025 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45158
I20240501 20:09:26.912986 11118 webserver.cc:460] Webserver started at http://127.10.219.130:41749/ using document root <none> and password file <none>
I20240501 20:09:26.913203 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:26.913278 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:26.913408 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:26.914058 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-1-root/instance:
uuid: "474c215a87764631b45692e1dd120180"
format_stamp: "Formatted at 2024-05-01 20:09:26 on dist-test-slave-n6pz"
I20240501 20:09:26.916930 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:26.919193 14164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:26.919675 11118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:26.919797 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-1-root
uuid: "474c215a87764631b45692e1dd120180"
format_stamp: "Formatted at 2024-05-01 20:09:26 on dist-test-slave-n6pz"
I20240501 20:09:26.919911 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:26.935222 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:26.935846 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:26.936407 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:26.937366 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:26.937427 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:26.937482 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:26.937528 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:26.950397 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.130:43315
I20240501 20:09:26.950425 14226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.130:43315 every 8 connection(s)
I20240501 20:09:26.952229 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:26.956885 14230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:26.959264 14231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:26.960409 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:26.960631 14227 heartbeater.cc:347] Connected to a master server at 127.10.219.190:42403
I20240501 20:09:26.960760 14227 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:26.961100 14227 heartbeater.cc:511] Master 127.10.219.190:42403 requested a full tablet report, sending...
W20240501 20:09:26.961475 14233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:26.962031 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:26.962116 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:26.962149 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594166962145 us; error 0 us; skew 500 ppm
I20240501 20:09:26.962250 14025 ts_manager.cc:194] Registered new tserver with Master: 474c215a87764631b45692e1dd120180 (127.10.219.130:43315)
I20240501 20:09:26.962338 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:26.963380 14025 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45174
I20240501 20:09:26.964056 11118 webserver.cc:460] Webserver started at http://127.10.219.131:43425/ using document root <none> and password file <none>
I20240501 20:09:26.964298 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:26.964394 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:26.964556 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:26.965250 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-2-root/instance:
uuid: "1a125645cb6447afb51002ad996604af"
format_stamp: "Formatted at 2024-05-01 20:09:26 on dist-test-slave-n6pz"
I20240501 20:09:26.968325 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:09:26.970666 14238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:26.971248 11118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:26.971387 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-2-root
uuid: "1a125645cb6447afb51002ad996604af"
format_stamp: "Formatted at 2024-05-01 20:09:26 on dist-test-slave-n6pz"
I20240501 20:09:26.971536 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714594084913313-11118-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:26.988135 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:26.988762 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:26.989374 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:26.997655 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:26.997752 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:26.997812 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:26.997843 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:27.012022 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.131:36775
I20240501 20:09:27.012042 14300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.131:36775 every 8 connection(s)
I20240501 20:09:27.018908 14301 heartbeater.cc:347] Connected to a master server at 127.10.219.190:42403
I20240501 20:09:27.019084 14301 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:27.019457 14301 heartbeater.cc:511] Master 127.10.219.190:42403 requested a full tablet report, sending...
I20240501 20:09:27.020444 14025 ts_manager.cc:194] Registered new tserver with Master: 1a125645cb6447afb51002ad996604af (127.10.219.131:36775)
I20240501 20:09:27.020927 11118 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008001012s
I20240501 20:09:27.021346 14025 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45190
I20240501 20:09:27.913805 14152 heartbeater.cc:503] Master 127.10.219.190:42403 was elected leader, sending a full tablet report...
I20240501 20:09:27.965040 14227 heartbeater.cc:503] Master 127.10.219.190:42403 was elected leader, sending a full tablet report...
I20240501 20:09:28.022848 14301 heartbeater.cc:503] Master 127.10.219.190:42403 was elected leader, sending a full tablet report...
I20240501 20:09:28.057029 11118 test_util.cc:268] Using random seed: 273820862
I20240501 20:09:28.070163 14025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45206:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:09:28.070894 14025 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:09:28.098585 14266 tablet_service.cc:1467] Processing CreateTablet for tablet f6f35c9c1c6b4388863825979b18e1ae (DEFAULT_TABLE table=test-workload [id=a0584014ebbb4c75964b2eebbbe91030]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:28.099170 14266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f35c9c1c6b4388863825979b18e1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.104290 14192 tablet_service.cc:1467] Processing CreateTablet for tablet f6f35c9c1c6b4388863825979b18e1ae (DEFAULT_TABLE table=test-workload [id=a0584014ebbb4c75964b2eebbbe91030]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:28.104807 14192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f35c9c1c6b4388863825979b18e1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.106680 14322 tablet_bootstrap.cc:492] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af: Bootstrap starting.
I20240501 20:09:28.107203 14117 tablet_service.cc:1467] Processing CreateTablet for tablet f6f35c9c1c6b4388863825979b18e1ae (DEFAULT_TABLE table=test-workload [id=a0584014ebbb4c75964b2eebbbe91030]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:28.107687 14117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f35c9c1c6b4388863825979b18e1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.108598 14322 tablet_bootstrap.cc:654] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:28.115022 14324 tablet_bootstrap.cc:492] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180: Bootstrap starting.
I20240501 20:09:28.116257 14325 tablet_bootstrap.cc:492] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f: Bootstrap starting.
I20240501 20:09:28.117256 14322 tablet_bootstrap.cc:492] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af: No bootstrap required, opened a new log
I20240501 20:09:28.117388 14324 tablet_bootstrap.cc:654] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:28.117542 14322 ts_tablet_manager.cc:1367] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20240501 20:09:28.118016 14325 tablet_bootstrap.cc:654] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:28.118602 14322 raft_consensus.cc:348] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a125645cb6447afb51002ad996604af" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36775 } } peers { permanent_uuid: "474c215a87764631b45692e1dd120180" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 43315 } } peers { permanent_uuid: "a11573fa0a8f4ba79f8534c441e1c15f" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41995 } }
I20240501 20:09:28.118952 14322 raft_consensus.cc:374] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:28.119033 14322 raft_consensus.cc:729] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a125645cb6447afb51002ad996604af, State: Initialized, Role: FOLLOWER
I20240501 20:09:28.119323 14322 consensus_queue.cc:260] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a125645cb6447afb51002ad996604af" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36775 } } peers { permanent_uuid: "474c215a87764631b45692e1dd120180" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 43315 } } peers { permanent_uuid: "a11573fa0a8f4ba79f8534c441e1c15f" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41995 } }
I20240501 20:09:28.119518 14324 tablet_bootstrap.cc:492] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180: No bootstrap required, opened a new log
I20240501 20:09:28.119681 14324 ts_tablet_manager.cc:1367] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s
I20240501 20:09:28.119710 14325 tablet_bootstrap.cc:492] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f: No bootstrap required, opened a new log
I20240501 20:09:28.119961 14325 ts_tablet_manager.cc:1367] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:28.120160 14322 ts_tablet_manager.cc:1398] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:28.120756 14325 raft_consensus.cc:348] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a125645cb6447afb51002ad996604af" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36775 } } peers { permanent_uuid: "474c215a87764631b45692e1dd120180" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 43315 } } peers { permanent_uuid: "a11573fa0a8f4ba79f8534c441e1c15f" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41995 } }
I20240501 20:09:28.120913 14324 raft_consensus.cc:348] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a125645cb6447afb51002ad996604af" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36775 } } peers { permanent_uuid: "474c215a87764631b45692e1dd120180" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 43315 } } peers { permanent_uuid: "a11573fa0a8f4ba79f8534c441e1c15f" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41995 } }
I20240501 20:09:28.121030 14325 raft_consensus.cc:374] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:28.121068 14324 raft_consensus.cc:374] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:28.121093 14325 raft_consensus.cc:729] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11573fa0a8f4ba79f8534c441e1c15f, State: Initialized, Role: FOLLOWER
I20240501 20:09:28.121147 14324 raft_consensus.cc:729] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474c215a87764631b45692e1dd120180, State: Initialized, Role: FOLLOWER
I20240501 20:09:28.121346 14324 consensus_queue.cc:260] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a125645cb6447afb51002ad996604af" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36775 } } peers { permanent_uuid: "474c215a87764631b45692e1dd120180" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 43315 } } peers { permanent_uuid: "a11573fa0a8f4ba79f8534c441e1c15f" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41995 } }
I20240501 20:09:28.121346 14325 consensus_queue.cc:260] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a125645cb6447afb51002ad996604af" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36775 } } peers { permanent_uuid: "474c215a87764631b45692e1dd120180" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 43315 } } peers { permanent_uuid: "a11573fa0a8f4ba79f8534c441e1c15f" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41995 } }
I20240501 20:09:28.122267 14324 ts_tablet_manager.cc:1398] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:28.122503 14325 ts_tablet_manager.cc:1398] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:28.442657 14328 raft_consensus.cc:482] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:28.442884 14328 raft_consensus.cc:504] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a125645cb6447afb51002ad996604af" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36775 } } peers { permanent_uuid: "474c215a87764631b45692e1dd120180" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 43315 } } peers { permanent_uuid: "a11573fa0a8f4ba79f8534c441e1c15f" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41995 } }
I20240501 20:09:28.443764 14328 leader_election.cc:288] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 474c215a87764631b45692e1dd120180 (127.10.219.130:43315), a11573fa0a8f4ba79f8534c441e1c15f (127.10.219.129:41995)
I20240501 20:09:28.449772 14127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f35c9c1c6b4388863825979b18e1ae" candidate_uuid: "1a125645cb6447afb51002ad996604af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11573fa0a8f4ba79f8534c441e1c15f" is_pre_election: true
I20240501 20:09:28.449704 14202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f35c9c1c6b4388863825979b18e1ae" candidate_uuid: "1a125645cb6447afb51002ad996604af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474c215a87764631b45692e1dd120180" is_pre_election: true
I20240501 20:09:28.450074 14202 raft_consensus.cc:2454] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a125645cb6447afb51002ad996604af in term 0.
I20240501 20:09:28.450074 14127 raft_consensus.cc:2454] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a125645cb6447afb51002ad996604af in term 0.
I20240501 20:09:28.450619 14240 leader_election.cc:302] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a125645cb6447afb51002ad996604af, a11573fa0a8f4ba79f8534c441e1c15f; no voters:
I20240501 20:09:28.450911 14328 raft_consensus.cc:2781] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:28.450986 14328 raft_consensus.cc:482] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:28.451035 14328 raft_consensus.cc:3037] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:28.453050 14328 raft_consensus.cc:504] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a125645cb6447afb51002ad996604af" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36775 } } peers { permanent_uuid: "474c215a87764631b45692e1dd120180" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 43315 } } peers { permanent_uuid: "a11573fa0a8f4ba79f8534c441e1c15f" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41995 } }
I20240501 20:09:28.453666 14328 leader_election.cc:288] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [CANDIDATE]: Term 1 election: Requested vote from peers 474c215a87764631b45692e1dd120180 (127.10.219.130:43315), a11573fa0a8f4ba79f8534c441e1c15f (127.10.219.129:41995)
I20240501 20:09:28.454226 14202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f35c9c1c6b4388863825979b18e1ae" candidate_uuid: "1a125645cb6447afb51002ad996604af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474c215a87764631b45692e1dd120180"
I20240501 20:09:28.454267 14127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f35c9c1c6b4388863825979b18e1ae" candidate_uuid: "1a125645cb6447afb51002ad996604af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11573fa0a8f4ba79f8534c441e1c15f"
I20240501 20:09:28.454455 14202 raft_consensus.cc:3037] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:28.454478 14127 raft_consensus.cc:3037] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:28.456471 14127 raft_consensus.cc:2454] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a125645cb6447afb51002ad996604af in term 1.
I20240501 20:09:28.456473 14202 raft_consensus.cc:2454] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a125645cb6447afb51002ad996604af in term 1.
I20240501 20:09:28.456964 14240 leader_election.cc:302] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a125645cb6447afb51002ad996604af, a11573fa0a8f4ba79f8534c441e1c15f; no voters:
I20240501 20:09:28.457266 14328 raft_consensus.cc:2781] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:28.457731 14328 raft_consensus.cc:686] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 1 LEADER]: Becoming Leader. State: Replica: 1a125645cb6447afb51002ad996604af, State: Running, Role: LEADER
I20240501 20:09:28.458001 14328 consensus_queue.cc:237] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a125645cb6447afb51002ad996604af" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36775 } } peers { permanent_uuid: "474c215a87764631b45692e1dd120180" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 43315 } } peers { permanent_uuid: "a11573fa0a8f4ba79f8534c441e1c15f" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41995 } }
I20240501 20:09:28.460857 14024 catalog_manager.cc:5484] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a125645cb6447afb51002ad996604af (127.10.219.131). New cstate: current_term: 1 leader_uuid: "1a125645cb6447afb51002ad996604af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a125645cb6447afb51002ad996604af" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 36775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "474c215a87764631b45692e1dd120180" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 43315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a11573fa0a8f4ba79f8534c441e1c15f" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41995 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:28.820164 14073 placement_policy_util.cc:407] tablet f6f35c9c1c6b4388863825979b18e1ae: detected majority of replicas (2 of 3) at location L0
I20240501 20:09:28.820799 14078 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:28.821213 14078 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:28.821319 14078 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:28.838526 14328 consensus_queue.cc:1035] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [LEADER]: Connected to new peer: Peer: permanent_uuid: "474c215a87764631b45692e1dd120180" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 43315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:28.843907 14328 consensus_queue.cc:1035] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a11573fa0a8f4ba79f8534c441e1c15f" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 41995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:29.823346 14073 placement_policy_util.cc:407] tablet f6f35c9c1c6b4388863825979b18e1ae: detected majority of replicas (2 of 3) at location L0
I20240501 20:09:30.821731 14078 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:30.822170 14078 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:30.822253 14078 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:30.825950 14073 placement_policy_util.cc:407] tablet f6f35c9c1c6b4388863825979b18e1ae: detected majority of replicas (2 of 3) at location L0
I20240501 20:09:31.828584 14073 placement_policy_util.cc:407] tablet f6f35c9c1c6b4388863825979b18e1ae: detected majority of replicas (2 of 3) at location L0
I20240501 20:09:32.822752 14078 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:32.823307 14078 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:32.823417 14078 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:32.831254 14073 placement_policy_util.cc:407] tablet f6f35c9c1c6b4388863825979b18e1ae: detected majority of replicas (2 of 3) at location L0
I20240501 20:09:33.833930 14073 placement_policy_util.cc:407] tablet f6f35c9c1c6b4388863825979b18e1ae: detected majority of replicas (2 of 3) at location L0
I20240501 20:09:34.823889 14078 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:34.824416 14078 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:34.824528 14078 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:34.836684 14073 placement_policy_util.cc:407] tablet f6f35c9c1c6b4388863825979b18e1ae: detected majority of replicas (2 of 3) at location L0
I20240501 20:09:35.839437 14073 placement_policy_util.cc:407] tablet f6f35c9c1c6b4388863825979b18e1ae: detected majority of replicas (2 of 3) at location L0
I20240501 20:09:36.824960 14078 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:36.825441 14078 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:36.825541 14078 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:36.842237 14073 placement_policy_util.cc:407] tablet f6f35c9c1c6b4388863825979b18e1ae: detected majority of replicas (2 of 3) at location L0
I20240501 20:09:37.845033 14073 placement_policy_util.cc:407] tablet f6f35c9c1c6b4388863825979b18e1ae: detected majority of replicas (2 of 3) at location L0
I20240501 20:09:38.826032 14078 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:38.826596 14078 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:38.826691 14078 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:38.847721 14073 placement_policy_util.cc:407] tablet f6f35c9c1c6b4388863825979b18e1ae: detected majority of replicas (2 of 3) at location L0
I20240501 20:09:39.524056 11118 tablet_server.cc:178] TabletServer@127.10.219.129:0 shutting down...
I20240501 20:09:39.540278 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:09:39.540633 11118 tablet_replica.cc:330] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f: stopping tablet replica
I20240501 20:09:39.540829 11118 raft_consensus.cc:2229] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.541023 11118 raft_consensus.cc:2258] T f6f35c9c1c6b4388863825979b18e1ae P a11573fa0a8f4ba79f8534c441e1c15f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.555424 11118 tablet_server.cc:194] TabletServer@127.10.219.129:0 shutdown complete.
I20240501 20:09:39.558993 11118 tablet_server.cc:178] TabletServer@127.10.219.130:0 shutting down...
I20240501 20:09:39.567329 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:09:39.567610 11118 tablet_replica.cc:330] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180: stopping tablet replica
I20240501 20:09:39.567813 11118 raft_consensus.cc:2229] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:39.568022 11118 raft_consensus.cc:2258] T f6f35c9c1c6b4388863825979b18e1ae P 474c215a87764631b45692e1dd120180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.582069 11118 tablet_server.cc:194] TabletServer@127.10.219.130:0 shutdown complete.
I20240501 20:09:39.585374 11118 tablet_server.cc:178] TabletServer@127.10.219.131:0 shutting down...
I20240501 20:09:39.594094 11118 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:09:39.594444 11118 tablet_replica.cc:330] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af: stopping tablet replica
I20240501 20:09:39.594641 11118 raft_consensus.cc:2229] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:39.594974 11118 raft_consensus.cc:2258] T f6f35c9c1c6b4388863825979b18e1ae P 1a125645cb6447afb51002ad996604af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.608997 11118 tablet_server.cc:194] TabletServer@127.10.219.131:0 shutdown complete.
I20240501 20:09:39.612293 11118 master.cc:537] Master@127.10.219.190:42403 shutting down...
I20240501 20:09:39.620644 11118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:39.620915 11118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:39.620999 11118 tablet_replica.cc:330] T 00000000000000000000000000000000 P b37000050a6f46ff97e7b0f4088acb3a: stopping tablet replica
I20240501 20:09:39.635010 11118 master.cc:552] Master@127.10.219.190:42403 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12926 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240501 20:09:39.658874 11118 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.10.219.190:41181
I20240501 20:09:39.659464 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:39.663425 14343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:39.664516 14344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:39.665817 14346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:39.666922 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:39.667204 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:39.667259 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:39.667284 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594179667281 us; error 0 us; skew 500 ppm
I20240501 20:09:39.667413 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:39.678941 11118 webserver.cc:460] Webserver started at http://127.10.219.190:41955/ using document root <none> and password file <none>
I20240501 20:09:39.679217 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:39.679278 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:39.679405 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:39.680095 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/master-0-root/instance:
uuid: "6c709e7e0eab4a4c82f65a7347cd0152"
format_stamp: "Formatted at 2024-05-01 20:09:39 on dist-test-slave-n6pz"
I20240501 20:09:39.682910 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 20:09:39.684636 14351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:39.685045 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:09:39.685154 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/master-0-root
uuid: "6c709e7e0eab4a4c82f65a7347cd0152"
format_stamp: "Formatted at 2024-05-01 20:09:39 on dist-test-slave-n6pz"
I20240501 20:09:39.685279 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:39.697288 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:39.697893 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:39.709846 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.190:41181
I20240501 20:09:39.709859 14402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.190:41181 every 8 connection(s)
I20240501 20:09:39.711313 14403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.715901 14403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152: Bootstrap starting.
I20240501 20:09:39.717145 14403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.718930 14403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152: No bootstrap required, opened a new log
I20240501 20:09:39.719570 14403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c709e7e0eab4a4c82f65a7347cd0152" member_type: VOTER }
I20240501 20:09:39.719713 14403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.719767 14403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c709e7e0eab4a4c82f65a7347cd0152, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.719970 14403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c709e7e0eab4a4c82f65a7347cd0152" member_type: VOTER }
I20240501 20:09:39.720084 14403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:39.720147 14403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:39.720232 14403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.722297 14403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c709e7e0eab4a4c82f65a7347cd0152" member_type: VOTER }
I20240501 20:09:39.722570 14403 leader_election.cc:302] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c709e7e0eab4a4c82f65a7347cd0152; no voters:
I20240501 20:09:39.723047 14403 leader_election.cc:288] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:39.723212 14406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:39.723744 14406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [term 1 LEADER]: Becoming Leader. State: Replica: 6c709e7e0eab4a4c82f65a7347cd0152, State: Running, Role: LEADER
I20240501 20:09:39.723924 14403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:09:39.723997 14406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c709e7e0eab4a4c82f65a7347cd0152" member_type: VOTER }
I20240501 20:09:39.724936 14408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c709e7e0eab4a4c82f65a7347cd0152. Latest consensus state: current_term: 1 leader_uuid: "6c709e7e0eab4a4c82f65a7347cd0152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c709e7e0eab4a4c82f65a7347cd0152" member_type: VOTER } }
I20240501 20:09:39.725111 14408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:39.724912 14407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c709e7e0eab4a4c82f65a7347cd0152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c709e7e0eab4a4c82f65a7347cd0152" member_type: VOTER } }
I20240501 20:09:39.725291 14407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152 [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:39.726488 14411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:09:39.727763 14411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:09:39.731590 14411 catalog_manager.cc:1347] Generated new cluster ID: 8b177d5d5d02453c82bd62e5ba42c999
I20240501 20:09:39.731705 14411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:09:39.731715 11118 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:09:39.750130 14411 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:09:39.751029 14411 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:09:39.761250 14411 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6c709e7e0eab4a4c82f65a7347cd0152: Generated new TSK 0
I20240501 20:09:39.761551 14411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:09:39.764469 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:39.768198 14424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:39.769104 14425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:39.770246 14427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:39.771238 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:39.771515 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:39.771575 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:39.771621 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594179771617 us; error 0 us; skew 500 ppm
I20240501 20:09:39.771775 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:39.773211 11118 webserver.cc:460] Webserver started at http://127.10.219.129:36957/ using document root <none> and password file <none>
I20240501 20:09:39.773468 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:39.773542 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:39.773682 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:39.774484 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-0-root/instance:
uuid: "8d368a895e2a48c8a17b384c0739e47c"
format_stamp: "Formatted at 2024-05-01 20:09:39 on dist-test-slave-n6pz"
I20240501 20:09:39.777362 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:39.779198 14432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:39.779572 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 20:09:39.779688 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-0-root
uuid: "8d368a895e2a48c8a17b384c0739e47c"
format_stamp: "Formatted at 2024-05-01 20:09:39 on dist-test-slave-n6pz"
I20240501 20:09:39.779814 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:39.788758 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:39.789399 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:39.789984 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:39.791014 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:39.791075 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:39.791154 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:39.791200 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:39.804509 14494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.129:33765 every 8 connection(s)
I20240501 20:09:39.804476 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.129:33765
I20240501 20:09:39.806362 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:39.810420 14500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:39.811280 14499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:39.812105 14502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:39.812825 14495 heartbeater.cc:347] Connected to a master server at 127.10.219.190:41181
I20240501 20:09:39.812995 14495 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:39.813195 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:39.813438 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:39.813493 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:39.813473 14495 heartbeater.cc:511] Master 127.10.219.190:41181 requested a full tablet report, sending...
I20240501 20:09:39.813571 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594179813567 us; error 0 us; skew 500 ppm
I20240501 20:09:39.813714 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:39.814643 14368 ts_manager.cc:194] Registered new tserver with Master: 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765)
I20240501 20:09:39.815222 11118 webserver.cc:460] Webserver started at http://127.10.219.130:36243/ using document root <none> and password file <none>
I20240501 20:09:39.815444 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:39.815538 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:39.815851 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:39.815968 14368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53958
I20240501 20:09:39.816790 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-1-root/instance:
uuid: "3da749f295434bceb714fb91c9598d45"
format_stamp: "Formatted at 2024-05-01 20:09:39 on dist-test-slave-n6pz"
I20240501 20:09:39.819703 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:09:39.821496 14507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:39.821861 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 20:09:39.821967 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-1-root
uuid: "3da749f295434bceb714fb91c9598d45"
format_stamp: "Formatted at 2024-05-01 20:09:39 on dist-test-slave-n6pz"
I20240501 20:09:39.822082 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:39.831523 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:39.832114 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:39.832744 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:39.833696 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:39.833755 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:39.833820 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:39.833868 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:39.848008 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.130:40971
I20240501 20:09:39.848032 14569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.130:40971 every 8 connection(s)
I20240501 20:09:39.849656 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:39.853595 14573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:39.854856 14574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:39.856055 14576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:39.856573 14570 heartbeater.cc:347] Connected to a master server at 127.10.219.190:41181
I20240501 20:09:39.856719 14570 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:39.856985 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:39.857247 14570 heartbeater.cc:511] Master 127.10.219.190:41181 requested a full tablet report, sending...
I20240501 20:09:39.857307 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:39.857389 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:39.857434 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594179857430 us; error 0 us; skew 500 ppm
I20240501 20:09:39.857592 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:39.858297 14368 ts_manager.cc:194] Registered new tserver with Master: 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:39.859198 11118 webserver.cc:460] Webserver started at http://127.10.219.131:36765/ using document root <none> and password file <none>
I20240501 20:09:39.859306 14368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53972
I20240501 20:09:39.859426 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:39.859505 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:39.859628 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:39.860495 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-2-root/instance:
uuid: "d66b9c8dab924baa8463d26f5f25b397"
format_stamp: "Formatted at 2024-05-01 20:09:39 on dist-test-slave-n6pz"
I20240501 20:09:39.863266 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:39.865278 14581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:39.865660 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 20:09:39.865773 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-2-root
uuid: "d66b9c8dab924baa8463d26f5f25b397"
format_stamp: "Formatted at 2024-05-01 20:09:39 on dist-test-slave-n6pz"
I20240501 20:09:39.865892 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:39.876792 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:39.877449 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:39.878062 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:39.878986 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:39.879050 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:39.879138 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:39.879182 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:39.893929 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.131:39403
I20240501 20:09:39.893956 14643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.131:39403 every 8 connection(s)
I20240501 20:09:39.901154 14644 heartbeater.cc:347] Connected to a master server at 127.10.219.190:41181
I20240501 20:09:39.901329 14644 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:39.901710 14644 heartbeater.cc:511] Master 127.10.219.190:41181 requested a full tablet report, sending...
I20240501 20:09:39.902717 14368 ts_manager.cc:194] Registered new tserver with Master: d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:39.903038 11118 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008027156s
I20240501 20:09:39.903524 14368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53980
I20240501 20:09:40.817715 14495 heartbeater.cc:503] Master 127.10.219.190:41181 was elected leader, sending a full tablet report...
I20240501 20:09:40.860863 14570 heartbeater.cc:503] Master 127.10.219.190:41181 was elected leader, sending a full tablet report...
I20240501 20:09:40.904907 14644 heartbeater.cc:503] Master 127.10.219.190:41181 was elected leader, sending a full tablet report...
I20240501 20:09:40.931885 11118 test_util.cc:268] Using random seed: 286695716
I20240501 20:09:40.947132 14368 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53994:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:09:40.948071 14368 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:09:40.975363 14609 tablet_service.cc:1467] Processing CreateTablet for tablet 45bd961f0ac04a4a812a818a864c902f (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 20:09:40.976146 14609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bd961f0ac04a4a812a818a864c902f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.979032 14608 tablet_service.cc:1467] Processing CreateTablet for tablet 7b90dffdaa47425883abd651fe391006 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 20:09:40.979851 14608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b90dffdaa47425883abd651fe391006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.982928 14607 tablet_service.cc:1467] Processing CreateTablet for tablet baf26068e4c246baa73afccb4e93c847 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 20:09:40.983567 14607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baf26068e4c246baa73afccb4e93c847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.984694 14605 tablet_service.cc:1467] Processing CreateTablet for tablet 85e1a32af97d4a238310403e907092e0 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 20:09:40.985177 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e1a32af97d4a238310403e907092e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.985880 14606 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf8ec6baf414c598a80753fe882d9a0 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 20:09:40.986478 14606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf8ec6baf414c598a80753fe882d9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.986474 14531 tablet_service.cc:1467] Processing CreateTablet for tablet 85e1a32af97d4a238310403e907092e0 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 20:09:40.986846 14533 tablet_service.cc:1467] Processing CreateTablet for tablet baf26068e4c246baa73afccb4e93c847 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 20:09:40.986980 14531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e1a32af97d4a238310403e907092e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.987342 14533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baf26068e4c246baa73afccb4e93c847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.989375 14534 tablet_service.cc:1467] Processing CreateTablet for tablet 7b90dffdaa47425883abd651fe391006 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 20:09:40.989835 14534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b90dffdaa47425883abd651fe391006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.986505 14535 tablet_service.cc:1467] Processing CreateTablet for tablet 45bd961f0ac04a4a812a818a864c902f (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 20:09:40.990943 14535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bd961f0ac04a4a812a818a864c902f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.994839 14532 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf8ec6baf414c598a80753fe882d9a0 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 20:09:40.995460 14532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf8ec6baf414c598a80753fe882d9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.000236 14665 tablet_bootstrap.cc:492] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45: Bootstrap starting.
I20240501 20:09:41.003417 14460 tablet_service.cc:1467] Processing CreateTablet for tablet 45bd961f0ac04a4a812a818a864c902f (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 20:09:41.004128 14460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bd961f0ac04a4a812a818a864c902f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.005139 14458 tablet_service.cc:1467] Processing CreateTablet for tablet baf26068e4c246baa73afccb4e93c847 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 20:09:41.005406 14664 tablet_bootstrap.cc:492] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397: Bootstrap starting.
I20240501 20:09:41.005677 14458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baf26068e4c246baa73afccb4e93c847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.006703 14665 tablet_bootstrap.cc:654] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.006925 14454 tablet_service.cc:1467] Processing CreateTablet for tablet f729e4d0e8dd4afbac82612b41d1eb4a (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 20:09:41.007447 14454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f729e4d0e8dd4afbac82612b41d1eb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.009053 14664 tablet_bootstrap.cc:654] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.010833 14457 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf8ec6baf414c598a80753fe882d9a0 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 20:09:41.011415 14457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf8ec6baf414c598a80753fe882d9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.011961 14604 tablet_service.cc:1467] Processing CreateTablet for tablet 4b3921d54966469c8de55002d0f93a10 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 20:09:41.012463 14604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b3921d54966469c8de55002d0f93a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.012679 14606 tablet_service.cc:1467] Processing CreateTablet for tablet b4ddb456073f4c4aa53517b561c948af (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 20:09:41.013116 14606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ddb456073f4c4aa53517b561c948af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.014961 14603 tablet_service.cc:1467] Processing CreateTablet for tablet 92c570e28c114771a13fcac4fa94af43 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 20:09:41.015648 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c570e28c114771a13fcac4fa94af43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.016047 14453 tablet_service.cc:1467] Processing CreateTablet for tablet cd5f05007317454bb4e7bdefd81ccd90 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 20:09:41.016551 14453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd5f05007317454bb4e7bdefd81ccd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.016764 14459 tablet_service.cc:1467] Processing CreateTablet for tablet 7b90dffdaa47425883abd651fe391006 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 20:09:41.017412 14459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b90dffdaa47425883abd651fe391006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.015641 14455 tablet_service.cc:1467] Processing CreateTablet for tablet b4ddb456073f4c4aa53517b561c948af (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 20:09:41.018895 14602 tablet_service.cc:1467] Processing CreateTablet for tablet 7c5cf239f44e48e2a85f97db902642ac (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 20:09:41.019218 14455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ddb456073f4c4aa53517b561c948af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.019421 14602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5cf239f44e48e2a85f97db902642ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.020946 14451 tablet_service.cc:1467] Processing CreateTablet for tablet 92c570e28c114771a13fcac4fa94af43 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 20:09:41.021669 14451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c570e28c114771a13fcac4fa94af43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.014961 14456 tablet_service.cc:1467] Processing CreateTablet for tablet 85e1a32af97d4a238310403e907092e0 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 20:09:41.022709 14456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e1a32af97d4a238310403e907092e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.022929 14601 tablet_service.cc:1467] Processing CreateTablet for tablet 9784e2a5d4324c63b42d41fe0fde222b (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 20:09:41.023408 14601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9784e2a5d4324c63b42d41fe0fde222b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.024724 14452 tablet_service.cc:1467] Processing CreateTablet for tablet 4b3921d54966469c8de55002d0f93a10 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 20:09:41.025491 14452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b3921d54966469c8de55002d0f93a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.026914 14531 tablet_service.cc:1467] Processing CreateTablet for tablet cd5f05007317454bb4e7bdefd81ccd90 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 20:09:41.027374 14531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd5f05007317454bb4e7bdefd81ccd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.030962 14530 tablet_service.cc:1467] Processing CreateTablet for tablet 4b3921d54966469c8de55002d0f93a10 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 20:09:41.031639 14530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b3921d54966469c8de55002d0f93a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.034922 14529 tablet_service.cc:1467] Processing CreateTablet for tablet 92c570e28c114771a13fcac4fa94af43 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 20:09:41.035615 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c570e28c114771a13fcac4fa94af43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.038934 14528 tablet_service.cc:1467] Processing CreateTablet for tablet 7c5cf239f44e48e2a85f97db902642ac (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 20:09:41.039672 14528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5cf239f44e48e2a85f97db902642ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.041714 14459 tablet_service.cc:1467] Processing CreateTablet for tablet 7c5cf239f44e48e2a85f97db902642ac (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 20:09:41.042245 14459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5cf239f44e48e2a85f97db902642ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.042862 14527 tablet_service.cc:1467] Processing CreateTablet for tablet 9784e2a5d4324c63b42d41fe0fde222b (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 20:09:41.042969 14451 tablet_service.cc:1467] Processing CreateTablet for tablet 9784e2a5d4324c63b42d41fe0fde222b (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 20:09:41.043447 14527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9784e2a5d4324c63b42d41fe0fde222b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.045061 14535 tablet_service.cc:1467] Processing CreateTablet for tablet b4ddb456073f4c4aa53517b561c948af (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 20:09:41.045749 14535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ddb456073f4c4aa53517b561c948af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.063163 14607 tablet_service.cc:1467] Processing CreateTablet for tablet f729e4d0e8dd4afbac82612b41d1eb4a (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 20:09:41.063884 14607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f729e4d0e8dd4afbac82612b41d1eb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.067852 14451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9784e2a5d4324c63b42d41fe0fde222b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.038966 14533 tablet_service.cc:1467] Processing CreateTablet for tablet f729e4d0e8dd4afbac82612b41d1eb4a (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 20:09:41.076507 14533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f729e4d0e8dd4afbac82612b41d1eb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.078976 14609 tablet_service.cc:1467] Processing CreateTablet for tablet cd5f05007317454bb4e7bdefd81ccd90 (DEFAULT_TABLE table=test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 20:09:41.081100 14609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd5f05007317454bb4e7bdefd81ccd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.093442 14664 tablet_bootstrap.cc:492] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397: No bootstrap required, opened a new log
I20240501 20:09:41.093686 14664 ts_tablet_manager.cc:1367] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397: Time spent bootstrapping tablet: real 0.088s user 0.002s sys 0.002s
I20240501 20:09:41.094640 14664 raft_consensus.cc:348] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.094920 14664 raft_consensus.cc:374] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.095027 14664 raft_consensus.cc:729] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.095355 14664 consensus_queue.cc:260] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.095511 14668 tablet_bootstrap.cc:492] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c: Bootstrap starting.
I20240501 20:09:41.095927 14665 tablet_bootstrap.cc:492] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45: No bootstrap required, opened a new log
I20240501 20:09:41.096163 14665 ts_tablet_manager.cc:1367] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45: Time spent bootstrapping tablet: real 0.096s user 0.000s sys 0.006s
I20240501 20:09:41.096357 14664 ts_tablet_manager.cc:1398] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:41.097126 14664 tablet_bootstrap.cc:492] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397: Bootstrap starting.
I20240501 20:09:41.097340 14665 raft_consensus.cc:348] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } }
I20240501 20:09:41.097680 14665 raft_consensus.cc:374] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.097826 14665 raft_consensus.cc:729] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.098033 14668 tablet_bootstrap.cc:654] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.098260 14665 consensus_queue.cc:260] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } }
I20240501 20:09:41.099202 14664 tablet_bootstrap.cc:654] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.103318 14665 ts_tablet_manager.cc:1398] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:09:41.104755 14668 tablet_bootstrap.cc:492] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c: No bootstrap required, opened a new log
I20240501 20:09:41.104497 14664 tablet_bootstrap.cc:492] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397: No bootstrap required, opened a new log
I20240501 20:09:41.104960 14668 ts_tablet_manager.cc:1367] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20240501 20:09:41.104967 14665 tablet_bootstrap.cc:492] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45: Bootstrap starting.
I20240501 20:09:41.105118 14664 ts_tablet_manager.cc:1367] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.007s
I20240501 20:09:41.106106 14668 raft_consensus.cc:348] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.106228 14664 raft_consensus.cc:348] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.106549 14668 raft_consensus.cc:374] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.106655 14664 raft_consensus.cc:374] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.106702 14668 raft_consensus.cc:729] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.106820 14664 raft_consensus.cc:729] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.107273 14665 tablet_bootstrap.cc:654] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.107120 14668 consensus_queue.cc:260] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.107167 14664 consensus_queue.cc:260] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.109369 14664 ts_tablet_manager.cc:1398] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:41.110002 14664 tablet_bootstrap.cc:492] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397: Bootstrap starting.
I20240501 20:09:41.113111 14664 tablet_bootstrap.cc:654] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.114683 14668 ts_tablet_manager.cc:1398] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20240501 20:09:41.116518 14668 tablet_bootstrap.cc:492] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c: Bootstrap starting.
I20240501 20:09:41.119392 14668 tablet_bootstrap.cc:654] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.122484 14665 tablet_bootstrap.cc:492] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45: No bootstrap required, opened a new log
I20240501 20:09:41.122795 14665 ts_tablet_manager.cc:1367] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.006s
I20240501 20:09:41.123226 14668 tablet_bootstrap.cc:492] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c: No bootstrap required, opened a new log
I20240501 20:09:41.123432 14668 ts_tablet_manager.cc:1367] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:09:41.123440 14664 tablet_bootstrap.cc:492] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397: No bootstrap required, opened a new log
I20240501 20:09:41.123728 14664 ts_tablet_manager.cc:1367] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240501 20:09:41.123849 14665 raft_consensus.cc:348] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.124133 14665 raft_consensus.cc:374] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.124291 14665 raft_consensus.cc:729] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.124389 14668 raft_consensus.cc:348] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.124576 14664 raft_consensus.cc:348] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } }
I20240501 20:09:41.124694 14668 raft_consensus.cc:374] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.124640 14665 consensus_queue.cc:260] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.124850 14664 raft_consensus.cc:374] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.124889 14668 raft_consensus.cc:729] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.124984 14664 raft_consensus.cc:729] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.125206 14668 consensus_queue.cc:260] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.125283 14664 consensus_queue.cc:260] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } }
I20240501 20:09:41.125922 14668 ts_tablet_manager.cc:1398] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:41.126367 14668 tablet_bootstrap.cc:492] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c: Bootstrap starting.
I20240501 20:09:41.127198 14664 ts_tablet_manager.cc:1398] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:41.127638 14664 tablet_bootstrap.cc:492] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397: Bootstrap starting.
I20240501 20:09:41.128768 14668 tablet_bootstrap.cc:654] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.129153 14665 ts_tablet_manager.cc:1398] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:41.130370 14665 tablet_bootstrap.cc:492] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45: Bootstrap starting.
I20240501 20:09:41.132565 14665 tablet_bootstrap.cc:654] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.134593 14664 tablet_bootstrap.cc:654] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.135560 14665 tablet_bootstrap.cc:492] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45: No bootstrap required, opened a new log
I20240501 20:09:41.135673 14668 tablet_bootstrap.cc:492] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c: No bootstrap required, opened a new log
I20240501 20:09:41.135792 14665 ts_tablet_manager.cc:1367] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 20:09:41.135888 14668 ts_tablet_manager.cc:1367] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20240501 20:09:41.136706 14665 raft_consensus.cc:348] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.136938 14665 raft_consensus.cc:374] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.137048 14665 raft_consensus.cc:729] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.137176 14668 raft_consensus.cc:348] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.137490 14668 raft_consensus.cc:374] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.137363 14665 consensus_queue.cc:260] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.137981 14665 ts_tablet_manager.cc:1398] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:41.138367 14665 tablet_bootstrap.cc:492] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45: Bootstrap starting.
I20240501 20:09:41.137624 14668 raft_consensus.cc:729] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.139629 14668 consensus_queue.cc:260] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.140331 14668 ts_tablet_manager.cc:1398] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:41.140326 14665 tablet_bootstrap.cc:654] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.140734 14668 tablet_bootstrap.cc:492] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c: Bootstrap starting.
I20240501 20:09:41.143177 14668 tablet_bootstrap.cc:654] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.145175 14664 tablet_bootstrap.cc:492] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397: No bootstrap required, opened a new log
I20240501 20:09:41.145560 14664 ts_tablet_manager.cc:1367] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.011s
I20240501 20:09:41.146811 14664 raft_consensus.cc:348] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.147311 14664 raft_consensus.cc:374] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.147496 14664 raft_consensus.cc:729] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.147964 14664 consensus_queue.cc:260] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.149006 14664 ts_tablet_manager.cc:1398] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:41.149559 14664 tablet_bootstrap.cc:492] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397: Bootstrap starting.
I20240501 20:09:41.152384 14664 tablet_bootstrap.cc:654] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.155061 14665 tablet_bootstrap.cc:492] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45: No bootstrap required, opened a new log
I20240501 20:09:41.155409 14665 ts_tablet_manager.cc:1367] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.015s
I20240501 20:09:41.156553 14665 raft_consensus.cc:348] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.156999 14665 raft_consensus.cc:374] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.157171 14668 tablet_bootstrap.cc:492] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c: No bootstrap required, opened a new log
I20240501 20:09:41.157173 14665 raft_consensus.cc:729] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.157542 14668 ts_tablet_manager.cc:1367] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20240501 20:09:41.157723 14665 consensus_queue.cc:260] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.158543 14665 ts_tablet_manager.cc:1398] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:41.158995 14665 tablet_bootstrap.cc:492] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45: Bootstrap starting.
I20240501 20:09:41.158891 14668 raft_consensus.cc:348] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.159327 14668 raft_consensus.cc:374] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.159484 14668 raft_consensus.cc:729] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.159925 14664 tablet_bootstrap.cc:492] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397: No bootstrap required, opened a new log
I20240501 20:09:41.159888 14668 consensus_queue.cc:260] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.160157 14664 ts_tablet_manager.cc:1367] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20240501 20:09:41.161345 14665 tablet_bootstrap.cc:654] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.160584 14668 ts_tablet_manager.cc:1398] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:41.161814 14664 raft_consensus.cc:348] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.162086 14668 tablet_bootstrap.cc:492] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c: Bootstrap starting.
I20240501 20:09:41.162169 14664 raft_consensus.cc:374] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.162366 14664 raft_consensus.cc:729] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.162699 14664 consensus_queue.cc:260] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.164089 14664 ts_tablet_manager.cc:1398] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:41.164152 14668 tablet_bootstrap.cc:654] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.164554 14664 tablet_bootstrap.cc:492] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397: Bootstrap starting.
I20240501 20:09:41.167354 14664 tablet_bootstrap.cc:654] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.176149 14665 tablet_bootstrap.cc:492] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45: No bootstrap required, opened a new log
I20240501 20:09:41.176539 14665 ts_tablet_manager.cc:1367] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.007s
I20240501 20:09:41.176832 14668 tablet_bootstrap.cc:492] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c: No bootstrap required, opened a new log
I20240501 20:09:41.177086 14668 ts_tablet_manager.cc:1367] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20240501 20:09:41.177434 14664 tablet_bootstrap.cc:492] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397: No bootstrap required, opened a new log
I20240501 20:09:41.177651 14664 ts_tablet_manager.cc:1367] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240501 20:09:41.177772 14665 raft_consensus.cc:348] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.178081 14665 raft_consensus.cc:374] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.177987 14668 raft_consensus.cc:348] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.178203 14665 raft_consensus.cc:729] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.178216 14668 raft_consensus.cc:374] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.178370 14668 raft_consensus.cc:729] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.178485 14664 raft_consensus.cc:348] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.178722 14664 raft_consensus.cc:374] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.178588 14665 consensus_queue.cc:260] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.178817 14664 raft_consensus.cc:729] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.178687 14668 consensus_queue.cc:260] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.179260 14665 ts_tablet_manager.cc:1398] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:41.179329 14668 ts_tablet_manager.cc:1398] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:41.179668 14668 tablet_bootstrap.cc:492] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c: Bootstrap starting.
I20240501 20:09:41.179764 14665 tablet_bootstrap.cc:492] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: Bootstrap starting.
I20240501 20:09:41.181381 14664 consensus_queue.cc:260] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.181938 14668 tablet_bootstrap.cc:654] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.182128 14673 raft_consensus.cc:482] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.182430 14673 raft_consensus.cc:504] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.183832 14673 leader_election.cc:288] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.185434 14664 ts_tablet_manager.cc:1398] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:09:41.186096 14670 raft_consensus.cc:482] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.186551 14670 raft_consensus.cc:504] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } }
I20240501 20:09:41.187330 14664 tablet_bootstrap.cc:492] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397: Bootstrap starting.
I20240501 20:09:41.188014 14670 leader_election.cc:288] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403), 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765)
I20240501 20:09:41.189954 14664 tablet_bootstrap.cc:654] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.190806 14668 tablet_bootstrap.cc:492] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c: No bootstrap required, opened a new log
I20240501 20:09:41.191113 14668 ts_tablet_manager.cc:1367] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 20:09:41.192346 14668 raft_consensus.cc:348] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.192831 14668 raft_consensus.cc:374] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.193006 14668 raft_consensus.cc:729] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.193456 14668 consensus_queue.cc:260] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.198536 14668 ts_tablet_manager.cc:1398] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:09:41.200801 14668 tablet_bootstrap.cc:492] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c: Bootstrap starting.
I20240501 20:09:41.182063 14665 tablet_bootstrap.cc:654] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.204407 14668 tablet_bootstrap.cc:654] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.206621 14664 tablet_bootstrap.cc:492] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397: No bootstrap required, opened a new log
I20240501 20:09:41.206984 14664 ts_tablet_manager.cc:1367] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.005s
I20240501 20:09:41.207419 14673 raft_consensus.cc:482] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.207651 14673 raft_consensus.cc:504] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.208722 14673 leader_election.cc:288] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.210191 14664 raft_consensus.cc:348] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.210682 14664 raft_consensus.cc:374] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.210855 14664 raft_consensus.cc:729] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.211303 14664 consensus_queue.cc:260] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.214071 14664 ts_tablet_manager.cc:1398] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:09:41.214751 14664 tablet_bootstrap.cc:492] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397: Bootstrap starting.
I20240501 20:09:41.217605 14664 tablet_bootstrap.cc:654] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.218151 14673 raft_consensus.cc:482] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.218400 14673 raft_consensus.cc:504] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.219413 14673 leader_election.cc:288] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.231170 14665 tablet_bootstrap.cc:492] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: No bootstrap required, opened a new log
I20240501 20:09:41.232154 14665 ts_tablet_manager.cc:1367] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.005s
I20240501 20:09:41.233819 14665 raft_consensus.cc:348] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.234278 14665 raft_consensus.cc:374] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.234472 14665 raft_consensus.cc:729] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.234915 14665 consensus_queue.cc:260] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.238646 14545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f729e4d0e8dd4afbac82612b41d1eb4a" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45" is_pre_election: true
I20240501 20:09:41.239387 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45" is_pre_election: true
I20240501 20:09:41.240058 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45" is_pre_election: true
W20240501 20:09:41.241768 14436 leader_election.cc:341] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:09:41.242127 14436 leader_election.cc:341] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:09:41.242476 14436 leader_election.cc:341] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:41.246246 14665 ts_tablet_manager.cc:1398] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240501 20:09:41.249958 14665 tablet_bootstrap.cc:492] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45: Bootstrap starting.
I20240501 20:09:41.253293 14665 tablet_bootstrap.cc:654] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.259039 14618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.260363 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f729e4d0e8dd4afbac82612b41d1eb4a" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.260692 14617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
W20240501 20:09:41.261739 14434 leader_election.cc:341] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:41.261955 14434 leader_election.cc:302] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c; no voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397
W20240501 20:09:41.262600 14434 leader_election.cc:341] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:41.262760 14434 leader_election.cc:302] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c; no voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397
I20240501 20:09:41.263307 14673 raft_consensus.cc:2726] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:41.263418 14696 raft_consensus.cc:2726] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:41.265640 14669 raft_consensus.cc:482] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.265841 14669 raft_consensus.cc:504] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.267138 14669 leader_election.cc:288] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
W20240501 20:09:41.269318 14434 leader_election.cc:341] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:41.269553 14434 leader_election.cc:302] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c; no voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397
I20240501 20:09:41.275133 14668 tablet_bootstrap.cc:492] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c: No bootstrap required, opened a new log
I20240501 20:09:41.275573 14668 ts_tablet_manager.cc:1367] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c: Time spent bootstrapping tablet: real 0.075s user 0.028s sys 0.042s
I20240501 20:09:41.275930 14669 raft_consensus.cc:482] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.276163 14669 raft_consensus.cc:504] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.276715 14668 raft_consensus.cc:348] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } }
I20240501 20:09:41.276911 14668 raft_consensus.cc:374] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.277011 14668 raft_consensus.cc:729] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.277107 14669 leader_election.cc:288] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:41.277283 14668 consensus_queue.cc:260] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } }
I20240501 20:09:41.277671 14673 raft_consensus.cc:2726] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:41.277941 14668 ts_tablet_manager.cc:1398] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:41.278268 14668 tablet_bootstrap.cc:492] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c: Bootstrap starting.
I20240501 20:09:41.279927 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bd961f0ac04a4a812a818a864c902f" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c" is_pre_election: true
I20240501 20:09:41.280352 14470 raft_consensus.cc:2454] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 0.
I20240501 20:09:41.280426 14668 tablet_bootstrap.cc:654] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.279927 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bd961f0ac04a4a812a818a864c902f" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.281143 14619 raft_consensus.cc:2454] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 0.
I20240501 20:09:41.281880 14509 leader_election.cc:302] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:41.282354 14670 raft_consensus.cc:2781] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.282476 14670 raft_consensus.cc:482] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.282574 14670 raft_consensus.cc:3037] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.283017 14665 tablet_bootstrap.cc:492] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45: No bootstrap required, opened a new log
I20240501 20:09:41.283190 14665 ts_tablet_manager.cc:1367] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.003s
I20240501 20:09:41.284104 14665 raft_consensus.cc:348] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.284332 14665 raft_consensus.cc:374] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.284420 14665 raft_consensus.cc:729] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.287129 14664 tablet_bootstrap.cc:492] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397: No bootstrap required, opened a new log
I20240501 20:09:41.287580 14664 ts_tablet_manager.cc:1367] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397: Time spent bootstrapping tablet: real 0.073s user 0.025s sys 0.013s
I20240501 20:09:41.288944 14665 consensus_queue.cc:260] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.289111 14664 raft_consensus.cc:348] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.291559 14696 raft_consensus.cc:482] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.291352 14670 raft_consensus.cc:504] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } }
I20240501 20:09:41.291733 14664 raft_consensus.cc:374] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.293790 14670 leader_election.cc:288] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Requested vote from peers d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403), 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765)
I20240501 20:09:41.294588 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bd961f0ac04a4a812a818a864c902f" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:41.294791 14664 raft_consensus.cc:729] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.294870 14619 raft_consensus.cc:3037] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.295336 14664 consensus_queue.cc:260] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.296188 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bd961f0ac04a4a812a818a864c902f" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:41.296474 14470 raft_consensus.cc:3037] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.297106 14696 raft_consensus.cc:504] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.298835 14664 ts_tablet_manager.cc:1398] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397: Time spent starting tablet: real 0.011s user 0.003s sys 0.001s
I20240501 20:09:41.299396 14673 raft_consensus.cc:482] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.299499 14664 tablet_bootstrap.cc:492] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397: Bootstrap starting.
I20240501 20:09:41.299605 14673 raft_consensus.cc:504] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.300434 14673 leader_election.cc:288] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.301280 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e1a32af97d4a238310403e907092e0" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45" is_pre_election: true
I20240501 20:09:41.301555 14543 raft_consensus.cc:2454] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 0.
I20240501 20:09:41.302399 14664 tablet_bootstrap.cc:654] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.302371 14436 leader_election.cc:302] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:41.303154 14673 raft_consensus.cc:2781] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.303323 14673 raft_consensus.cc:482] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.303481 14673 raft_consensus.cc:3037] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.303709 14617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e1a32af97d4a238310403e907092e0" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.304014 14617 raft_consensus.cc:2454] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 0.
I20240501 20:09:41.306478 14619 raft_consensus.cc:2454] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 1.
I20240501 20:09:41.307966 14509 leader_election.cc:302] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:41.308478 14670 raft_consensus.cc:2781] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.308394 14673 raft_consensus.cc:504] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.308637 14670 raft_consensus.cc:686] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:41.308954 14670 consensus_queue.cc:237] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } }
I20240501 20:09:41.309285 14470 raft_consensus.cc:2454] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 1.
I20240501 20:09:41.310266 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e1a32af97d4a238310403e907092e0" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:41.310688 14543 raft_consensus.cc:3037] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.310339 14696 leader_election.cc:288] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.311416 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.311679 14619 raft_consensus.cc:2454] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 0.
I20240501 20:09:41.312146 14668 tablet_bootstrap.cc:492] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c: No bootstrap required, opened a new log
I20240501 20:09:41.312451 14668 ts_tablet_manager.cc:1367] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.007s
I20240501 20:09:41.313880 14668 raft_consensus.cc:348] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.314188 14668 raft_consensus.cc:374] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.314345 14668 raft_consensus.cc:729] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.314888 14668 consensus_queue.cc:260] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.315260 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45" is_pre_election: true
I20240501 20:09:41.315680 14544 raft_consensus.cc:2454] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 0.
I20240501 20:09:41.315773 14668 ts_tablet_manager.cc:1398] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20240501 20:09:41.316215 14668 tablet_bootstrap.cc:492] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c: Bootstrap starting.
I20240501 20:09:41.317996 14436 leader_election.cc:302] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:41.318660 14668 tablet_bootstrap.cc:654] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.320129 14696 raft_consensus.cc:2781] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.320287 14696 raft_consensus.cc:482] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.320434 14696 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.324551 14696 raft_consensus.cc:504] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.327466 14543 raft_consensus.cc:2454] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 1.
I20240501 20:09:41.330016 14673 leader_election.cc:288] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.332860 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e1a32af97d4a238310403e907092e0" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:41.333250 14619 raft_consensus.cc:3037] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.335067 14436 leader_election.cc:302] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:41.335469 14673 raft_consensus.cc:2781] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.335662 14673 raft_consensus.cc:686] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Becoming Leader. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:41.335768 14665 ts_tablet_manager.cc:1398] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45: Time spent starting tablet: real 0.052s user 0.018s sys 0.028s
I20240501 20:09:41.335983 14673 consensus_queue.cc:237] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.338235 14619 raft_consensus.cc:2454] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 1.
I20240501 20:09:41.339427 14665 tablet_bootstrap.cc:492] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45: Bootstrap starting.
I20240501 20:09:41.343092 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:41.343428 14543 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.344507 14665 tablet_bootstrap.cc:654] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.344674 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c" is_pre_election: true
I20240501 20:09:41.345229 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c" is_pre_election: true
I20240501 20:09:41.345671 14469 raft_consensus.cc:2379] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66b9c8dab924baa8463d26f5f25b397 in current term 1: Already voted for candidate 8d368a895e2a48c8a17b384c0739e47c in this term.
I20240501 20:09:41.346855 14368 catalog_manager.cc:5484] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:41.347890 14543 raft_consensus.cc:2454] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 1.
I20240501 20:09:41.348989 14436 leader_election.cc:302] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:41.349687 14673 raft_consensus.cc:2781] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.350024 14673 raft_consensus.cc:686] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Becoming Leader. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:41.350652 14673 consensus_queue.cc:237] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.351177 14664 tablet_bootstrap.cc:492] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397: No bootstrap required, opened a new log
I20240501 20:09:41.351375 14664 ts_tablet_manager.cc:1367] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.009s
I20240501 20:09:41.352249 14664 raft_consensus.cc:348] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.352530 14664 raft_consensus.cc:374] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.352653 14664 raft_consensus.cc:729] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.353557 14696 leader_election.cc:288] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.353094 14367 catalog_manager.cc:5484] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129). New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.355626 14664 consensus_queue.cc:260] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.358100 14664 ts_tablet_manager.cc:1398] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240501 20:09:41.358566 14664 tablet_bootstrap.cc:492] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397: Bootstrap starting.
I20240501 20:09:41.360662 14664 tablet_bootstrap.cc:654] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.364827 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:41.365217 14619 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.367305 14664 tablet_bootstrap.cc:492] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397: No bootstrap required, opened a new log
I20240501 20:09:41.367570 14664 ts_tablet_manager.cc:1367] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 20:09:41.368643 14619 raft_consensus.cc:2454] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 1.
W20240501 20:09:41.369192 14583 leader_election.cc:341] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:41.369946 14670 raft_consensus.cc:482] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.370170 14670 raft_consensus.cc:504] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.368484 14664 raft_consensus.cc:348] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.370750 14664 raft_consensus.cc:374] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.370867 14664 raft_consensus.cc:729] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.371157 14670 leader_election.cc:288] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.371513 14664 consensus_queue.cc:260] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.371894 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c" is_pre_election: true
I20240501 20:09:41.372180 14469 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 0.
I20240501 20:09:41.372179 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.372819 14509 leader_election.cc:302] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
W20240501 20:09:41.373306 14509 leader_election.cc:341] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:41.373638 14664 ts_tablet_manager.cc:1398] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240501 20:09:41.373845 14701 raft_consensus.cc:2781] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.374017 14701 raft_consensus.cc:482] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.374187 14701 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.376303 14668 tablet_bootstrap.cc:492] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c: No bootstrap required, opened a new log
I20240501 20:09:41.376560 14668 ts_tablet_manager.cc:1367] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c: Time spent bootstrapping tablet: real 0.060s user 0.018s sys 0.023s
I20240501 20:09:41.377511 14668 raft_consensus.cc:348] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.377812 14668 raft_consensus.cc:374] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.377735 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45" is_pre_election: true
I20240501 20:09:41.377925 14668 raft_consensus.cc:729] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.378213 14668 consensus_queue.cc:260] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.378360 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45" is_pre_election: true
I20240501 20:09:41.378772 14544 raft_consensus.cc:2379] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66b9c8dab924baa8463d26f5f25b397 in current term 1: Already voted for candidate 8d368a895e2a48c8a17b384c0739e47c in this term.
I20240501 20:09:41.378999 14668 ts_tablet_manager.cc:1398] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:41.379479 14668 tablet_bootstrap.cc:492] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c: Bootstrap starting.
W20240501 20:09:41.379493 14585 leader_election.cc:341] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:41.379760 14585 leader_election.cc:302] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66b9c8dab924baa8463d26f5f25b397; no voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:41.380615 14585 leader_election.cc:302] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66b9c8dab924baa8463d26f5f25b397; no voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:41.381358 14669 raft_consensus.cc:2726] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:41.381623 14669 raft_consensus.cc:2726] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:41.383060 14701 raft_consensus.cc:504] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.384860 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:41.385221 14469 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.390183 14469 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 1.
I20240501 20:09:41.374027 14664 tablet_bootstrap.cc:492] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397: Bootstrap starting.
I20240501 20:09:41.390931 14665 tablet_bootstrap.cc:492] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45: No bootstrap required, opened a new log
I20240501 20:09:41.390942 14509 leader_election.cc:302] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:41.391160 14665 ts_tablet_manager.cc:1367] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45: Time spent bootstrapping tablet: real 0.052s user 0.014s sys 0.018s
I20240501 20:09:41.391438 14670 raft_consensus.cc:2781] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.392099 14665 raft_consensus.cc:348] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.392364 14665 raft_consensus.cc:374] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.392457 14665 raft_consensus.cc:729] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.392280 14368 catalog_manager.cc:5484] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129). New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.392753 14665 consensus_queue.cc:260] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.395323 14701 leader_election.cc:288] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.396030 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:41.398114 14713 raft_consensus.cc:482] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.399003 14664 tablet_bootstrap.cc:654] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397: Neither blocks nor log segments found. Creating new log.
W20240501 20:09:41.399286 14509 leader_election.cc:341] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:41.398824 14713 raft_consensus.cc:504] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.401189 14665 ts_tablet_manager.cc:1398] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45: Time spent starting tablet: real 0.010s user 0.002s sys 0.006s
I20240501 20:09:41.398561 14668 tablet_bootstrap.cc:654] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.401589 14665 tablet_bootstrap.cc:492] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45: Bootstrap starting.
I20240501 20:09:41.402194 14670 raft_consensus.cc:686] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:41.402768 14713 leader_election.cc:288] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.403807 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.404107 14665 tablet_bootstrap.cc:654] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.404175 14619 raft_consensus.cc:2454] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 0.
I20240501 20:09:41.404907 14509 leader_election.cc:302] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:41.405711 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c" is_pre_election: true
I20240501 20:09:41.406049 14469 raft_consensus.cc:2454] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 0.
I20240501 20:09:41.407029 14670 consensus_queue.cc:237] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.408025 14713 raft_consensus.cc:2781] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.408183 14713 raft_consensus.cc:482] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.408329 14713 raft_consensus.cc:3037] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.410924 14668 tablet_bootstrap.cc:492] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c: No bootstrap required, opened a new log
I20240501 20:09:41.411250 14668 ts_tablet_manager.cc:1367] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.009s
I20240501 20:09:41.411922 14713 raft_consensus.cc:504] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.412506 14668 raft_consensus.cc:348] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.412793 14668 raft_consensus.cc:374] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.412919 14668 raft_consensus.cc:729] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.413219 14668 consensus_queue.cc:260] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.413375 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:41.413638 14469 raft_consensus.cc:3037] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.414090 14668 ts_tablet_manager.cc:1398] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:41.414144 14665 tablet_bootstrap.cc:492] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45: No bootstrap required, opened a new log
I20240501 20:09:41.414512 14665 ts_tablet_manager.cc:1367] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20240501 20:09:41.414563 14668 tablet_bootstrap.cc:492] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c: Bootstrap starting.
I20240501 20:09:41.415670 14665 raft_consensus.cc:348] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.416069 14665 raft_consensus.cc:374] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.416272 14665 raft_consensus.cc:729] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.416612 14665 consensus_queue.cc:260] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.416951 14668 tablet_bootstrap.cc:654] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.417373 14665 ts_tablet_manager.cc:1398] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:41.417428 14469 raft_consensus.cc:2454] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 1.
I20240501 20:09:41.417866 14665 tablet_bootstrap.cc:492] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45: Bootstrap starting.
I20240501 20:09:41.418032 14509 leader_election.cc:302] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:41.418529 14711 raft_consensus.cc:2781] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.418764 14711 raft_consensus.cc:686] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:41.419337 14711 consensus_queue.cc:237] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.420559 14701 raft_consensus.cc:482] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.420588 14665 tablet_bootstrap.cc:654] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.420763 14701 raft_consensus.cc:504] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.421629 14701 leader_election.cc:288] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.422626 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c" is_pre_election: true
I20240501 20:09:41.423048 14470 raft_consensus.cc:2454] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 0.
I20240501 20:09:41.422636 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.423764 14619 raft_consensus.cc:2454] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 0.
I20240501 20:09:41.424795 14509 leader_election.cc:302] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:41.425680 14701 raft_consensus.cc:2781] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.425858 14701 raft_consensus.cc:482] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.426029 14701 raft_consensus.cc:3037] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.427780 14673 raft_consensus.cc:482] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.428027 14673 raft_consensus.cc:504] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.428901 14673 leader_election.cc:288] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.429826 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baf26068e4c246baa73afccb4e93c847" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.430116 14619 raft_consensus.cc:2454] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 0.
I20240501 20:09:41.430156 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baf26068e4c246baa73afccb4e93c847" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45" is_pre_election: true
I20240501 20:09:41.430466 14544 raft_consensus.cc:2454] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 0.
I20240501 20:09:41.430824 14434 leader_election.cc:302] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:41.431166 14673 raft_consensus.cc:2781] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.431308 14673 raft_consensus.cc:482] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.431488 14673 raft_consensus.cc:3037] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.431303 14701 raft_consensus.cc:504] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.432140 14701 leader_election.cc:288] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.433220 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:41.433538 14470 raft_consensus.cc:3037] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.434713 14673 raft_consensus.cc:504] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.436118 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baf26068e4c246baa73afccb4e93c847" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:41.436400 14544 raft_consensus.cc:3037] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.436619 14470 raft_consensus.cc:2454] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 1.
I20240501 20:09:41.437427 14509 leader_election.cc:302] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:41.437427 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:41.437945 14619 raft_consensus.cc:3037] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.440048 14544 raft_consensus.cc:2454] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 1.
I20240501 20:09:41.440650 14436 leader_election.cc:302] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:41.440863 14619 raft_consensus.cc:2454] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 1.
I20240501 20:09:41.441062 14696 raft_consensus.cc:2781] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.441267 14696 raft_consensus.cc:686] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Becoming Leader. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:41.441606 14696 consensus_queue.cc:237] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.443563 14701 raft_consensus.cc:2781] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.443987 14701 raft_consensus.cc:686] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:41.444206 14673 leader_election.cc:288] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.445011 14701 consensus_queue.cc:237] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.446743 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baf26068e4c246baa73afccb4e93c847" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:41.447124 14619 raft_consensus.cc:3037] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.448422 14701 raft_consensus.cc:482] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.448701 14701 raft_consensus.cc:504] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.450428 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c" is_pre_election: true
W20240501 20:09:41.451308 14509 leader_election.cc:341] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:41.452807 14617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.453169 14617 raft_consensus.cc:2454] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 0.
I20240501 20:09:41.453435 14619 raft_consensus.cc:2454] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 1.
I20240501 20:09:41.454372 14509 leader_election.cc:302] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters: 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:41.454800 14712 raft_consensus.cc:2781] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.454947 14712 raft_consensus.cc:482] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.455101 14712 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.458025 14368 catalog_manager.cc:5484] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129). New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.458570 14712 raft_consensus.cc:504] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.459386 14712 leader_election.cc:288] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.460486 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
W20240501 20:09:41.461174 14509 leader_election.cc:341] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:41.461539 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:41.461766 14619 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.465942 14619 raft_consensus.cc:2454] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 1.
I20240501 20:09:41.473887 14509 leader_election.cc:302] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters: 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:41.474828 14711 raft_consensus.cc:2781] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.475117 14711 raft_consensus.cc:686] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:41.475610 14711 consensus_queue.cc:237] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.467288 14701 leader_election.cc:288] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.472901 14368 catalog_manager.cc:5484] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.479928 14664 tablet_bootstrap.cc:492] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397: No bootstrap required, opened a new log
I20240501 20:09:41.480293 14664 ts_tablet_manager.cc:1367] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397: Time spent bootstrapping tablet: real 0.106s user 0.028s sys 0.044s
I20240501 20:09:41.481758 14664 raft_consensus.cc:348] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.482147 14664 raft_consensus.cc:374] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.483886 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:41.484278 14619 raft_consensus.cc:3037] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.486908 14713 leader_election.cc:288] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.491659 14619 raft_consensus.cc:2454] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 1.
I20240501 20:09:41.492723 14664 raft_consensus.cc:729] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.492909 14368 catalog_manager.cc:5484] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.493239 14664 consensus_queue.cc:260] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.494112 14664 ts_tablet_manager.cc:1398] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 20:09:41.493798 14368 catalog_manager.cc:5484] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.494890 14368 catalog_manager.cc:5484] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.495296 14664 tablet_bootstrap.cc:492] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397: Bootstrap starting.
I20240501 20:09:41.497174 14665 tablet_bootstrap.cc:492] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45: No bootstrap required, opened a new log
I20240501 20:09:41.497485 14665 ts_tablet_manager.cc:1367] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45: Time spent bootstrapping tablet: real 0.080s user 0.020s sys 0.036s
I20240501 20:09:41.497202 14664 tablet_bootstrap.cc:654] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.498399 14668 tablet_bootstrap.cc:492] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c: No bootstrap required, opened a new log
I20240501 20:09:41.498850 14668 ts_tablet_manager.cc:1367] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c: Time spent bootstrapping tablet: real 0.084s user 0.030s sys 0.016s
I20240501 20:09:41.498394 14665 raft_consensus.cc:348] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.500180 14668 raft_consensus.cc:348] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.500546 14668 raft_consensus.cc:374] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.500698 14668 raft_consensus.cc:729] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.500707 14665 raft_consensus.cc:374] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.501004 14665 raft_consensus.cc:729] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.501015 14668 consensus_queue.cc:260] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.501667 14668 ts_tablet_manager.cc:1398] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:41.501729 14664 tablet_bootstrap.cc:492] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397: No bootstrap required, opened a new log
I20240501 20:09:41.501964 14664 ts_tablet_manager.cc:1367] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240501 20:09:41.501973 14668 tablet_bootstrap.cc:492] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c: Bootstrap starting.
I20240501 20:09:41.502127 14665 consensus_queue.cc:260] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.502756 14665 ts_tablet_manager.cc:1398] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:41.503144 14665 tablet_bootstrap.cc:492] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45: Bootstrap starting.
I20240501 20:09:41.503183 14664 raft_consensus.cc:348] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.505003 14665 tablet_bootstrap.cc:654] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.510084 14668 tablet_bootstrap.cc:654] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.511366 14664 raft_consensus.cc:374] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.511485 14664 raft_consensus.cc:729] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.511751 14664 consensus_queue.cc:260] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.511987 14665 tablet_bootstrap.cc:492] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45: No bootstrap required, opened a new log
I20240501 20:09:41.512212 14665 ts_tablet_manager.cc:1367] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20240501 20:09:41.512326 14664 ts_tablet_manager.cc:1398] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240501 20:09:41.513242 14665 raft_consensus.cc:348] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.513530 14665 raft_consensus.cc:374] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.513618 14665 raft_consensus.cc:729] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.513911 14665 consensus_queue.cc:260] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.514792 14665 ts_tablet_manager.cc:1398] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 20:09:41.515173 14665 tablet_bootstrap.cc:492] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45: Bootstrap starting.
I20240501 20:09:41.517458 14665 tablet_bootstrap.cc:654] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.519829 14668 tablet_bootstrap.cc:492] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c: No bootstrap required, opened a new log
I20240501 20:09:41.519944 14665 tablet_bootstrap.cc:492] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45: No bootstrap required, opened a new log
I20240501 20:09:41.520007 14668 ts_tablet_manager.cc:1367] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20240501 20:09:41.520093 14665 ts_tablet_manager.cc:1367] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:09:41.520717 14665 raft_consensus.cc:348] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.520771 14668 raft_consensus.cc:348] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.520917 14665 raft_consensus.cc:374] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.520936 14668 raft_consensus.cc:374] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.521008 14665 raft_consensus.cc:729] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.521046 14668 raft_consensus.cc:729] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.521256 14665 consensus_queue.cc:260] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.521293 14668 consensus_queue.cc:260] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.521816 14665 ts_tablet_manager.cc:1398] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:41.521826 14668 ts_tablet_manager.cc:1398] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:41.555785 14696 raft_consensus.cc:482] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.556015 14696 raft_consensus.cc:504] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.556689 14696 leader_election.cc:288] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.557327 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.557327 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45" is_pre_election: true
I20240501 20:09:41.557641 14619 raft_consensus.cc:2379] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d368a895e2a48c8a17b384c0739e47c in current term 1: Already voted for candidate 3da749f295434bceb714fb91c9598d45 in this term.
I20240501 20:09:41.558101 14436 leader_election.cc:302] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c; no voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397
I20240501 20:09:41.558426 14696 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.560406 14696 raft_consensus.cc:2726] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:41.580538 14711 raft_consensus.cc:482] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.580777 14711 raft_consensus.cc:504] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.581640 14711 leader_election.cc:288] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.582243 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c" is_pre_election: true
I20240501 20:09:41.582243 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.582556 14469 raft_consensus.cc:2454] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 0.
I20240501 20:09:41.582556 14619 raft_consensus.cc:2454] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 0.
I20240501 20:09:41.583009 14509 leader_election.cc:302] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:41.583279 14711 raft_consensus.cc:2781] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.583376 14711 raft_consensus.cc:482] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.583426 14711 raft_consensus.cc:3037] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.585547 14711 raft_consensus.cc:504] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.586210 14711 leader_election.cc:288] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.586853 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:41.586905 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:41.587033 14469 raft_consensus.cc:3037] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.587057 14619 raft_consensus.cc:3037] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.589131 14619 raft_consensus.cc:2454] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 1.
I20240501 20:09:41.589133 14469 raft_consensus.cc:2454] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 1.
I20240501 20:09:41.589633 14509 leader_election.cc:302] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:41.589922 14711 raft_consensus.cc:2781] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.590066 14711 raft_consensus.cc:686] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:41.590379 14711 consensus_queue.cc:237] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.593340 14366 catalog_manager.cc:5484] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.699362 14710 raft_consensus.cc:482] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.699611 14710 raft_consensus.cc:504] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.700268 14710 leader_election.cc:288] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:41.700937 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45" is_pre_election: true
I20240501 20:09:41.700937 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c" is_pre_election: true
I20240501 20:09:41.701256 14544 raft_consensus.cc:2454] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 0.
I20240501 20:09:41.701256 14469 raft_consensus.cc:2454] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 0.
I20240501 20:09:41.701792 14585 leader_election.cc:302] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:41.702137 14710 raft_consensus.cc:2781] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.702234 14710 raft_consensus.cc:482] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.702353 14710 raft_consensus.cc:3037] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.704483 14710 raft_consensus.cc:504] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.705132 14710 leader_election.cc:288] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 1 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:41.705783 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:41.706044 14469 raft_consensus.cc:3037] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.705783 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:41.706418 14544 raft_consensus.cc:3037] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.709240 14544 raft_consensus.cc:2454] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 1.
I20240501 20:09:41.709244 14469 raft_consensus.cc:2454] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 1.
I20240501 20:09:41.709808 14585 leader_election.cc:302] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:41.710129 14710 raft_consensus.cc:2781] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.710287 14710 raft_consensus.cc:686] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 1 LEADER]: Becoming Leader. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:41.710605 14710 consensus_queue.cc:237] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.712795 14711 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.713589 14366 catalog_manager.cc:5484] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: term changed from 0 to 1, leader changed from <none> to d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131). New cstate: current_term: 1 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:41.714685 14696 raft_consensus.cc:482] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.714848 14696 raft_consensus.cc:504] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.715502 14696 leader_election.cc:288] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.715849 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5cf239f44e48e2a85f97db902642ac" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45" is_pre_election: true
I20240501 20:09:41.716012 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5cf239f44e48e2a85f97db902642ac" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.716064 14544 raft_consensus.cc:2454] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 0.
I20240501 20:09:41.716181 14619 raft_consensus.cc:2454] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 0.
I20240501 20:09:41.716457 14436 leader_election.cc:302] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:41.716950 14696 raft_consensus.cc:2781] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.717052 14696 raft_consensus.cc:482] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.717140 14696 raft_consensus.cc:3037] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.719944 14711 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.720853 14696 raft_consensus.cc:504] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.721360 14669 raft_consensus.cc:482] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.721490 14669 raft_consensus.cc:504] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.721959 14617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5cf239f44e48e2a85f97db902642ac" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:41.722134 14617 raft_consensus.cc:3037] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.722177 14669 leader_election.cc:288] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:41.722757 14696 leader_election.cc:288] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.722731 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5cf239f44e48e2a85f97db902642ac" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c" is_pre_election: true
I20240501 20:09:41.722926 14470 raft_consensus.cc:2379] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66b9c8dab924baa8463d26f5f25b397 in current term 1: Already voted for candidate 8d368a895e2a48c8a17b384c0739e47c in this term.
I20240501 20:09:41.723526 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5cf239f44e48e2a85f97db902642ac" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:41.723752 14543 raft_consensus.cc:3037] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.724820 14617 raft_consensus.cc:2454] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 1.
I20240501 20:09:41.725118 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5cf239f44e48e2a85f97db902642ac" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da749f295434bceb714fb91c9598d45" is_pre_election: true
I20240501 20:09:41.725546 14434 leader_election.cc:302] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:41.725615 14585 leader_election.cc:302] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66b9c8dab924baa8463d26f5f25b397; no voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:41.725847 14696 raft_consensus.cc:2781] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.725952 14696 raft_consensus.cc:686] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Becoming Leader. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:41.725991 14710 raft_consensus.cc:2726] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:41.726197 14696 consensus_queue.cc:237] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.727556 14543 raft_consensus.cc:2454] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 1.
I20240501 20:09:41.727726 14713 raft_consensus.cc:482] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.727872 14713 raft_consensus.cc:504] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.728403 14713 leader_election.cc:288] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.728958 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f729e4d0e8dd4afbac82612b41d1eb4a" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c" is_pre_election: true
I20240501 20:09:41.728928 14617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f729e4d0e8dd4afbac82612b41d1eb4a" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397" is_pre_election: true
I20240501 20:09:41.729142 14469 raft_consensus.cc:2454] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 0.
I20240501 20:09:41.729156 14617 raft_consensus.cc:2454] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 0.
I20240501 20:09:41.729524 14509 leader_election.cc:302] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:41.729851 14713 raft_consensus.cc:2781] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.729908 14713 raft_consensus.cc:482] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.729948 14713 raft_consensus.cc:3037] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.729960 14366 catalog_manager.cc:5484] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129). New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.731618 14422 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:41.732798 14713 raft_consensus.cc:504] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.733309 14713 leader_election.cc:288] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:41.733780 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f729e4d0e8dd4afbac82612b41d1eb4a" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:41.733909 14422 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:41.733918 14469 raft_consensus.cc:3037] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.734014 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f729e4d0e8dd4afbac82612b41d1eb4a" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:41.734136 14619 raft_consensus.cc:3037] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:09:41.734798 14416 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:09:41.737088 14619 raft_consensus.cc:2454] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 1.
I20240501 20:09:41.737088 14469 raft_consensus.cc:2454] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 1.
I20240501 20:09:41.737519 14509 leader_election.cc:302] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:41.738845 14713 raft_consensus.cc:2781] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.739004 14713 raft_consensus.cc:686] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:41.739239 14713 consensus_queue.cc:237] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:41.742120 14368 catalog_manager.cc:5484] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.744457 14696 consensus_queue.cc:1035] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.749058 14738 consensus_queue.cc:1035] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.819161 14711 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.823640 14619 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.828418 14712 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.832896 14738 consensus_queue.cc:1035] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.840960 14703 consensus_queue.cc:1035] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.845361 14701 consensus_queue.cc:1035] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.845304 14696 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:09:41.861496 14738 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.866415 14701 consensus_queue.cc:1035] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.871181 14701 consensus_queue.cc:1035] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.877274 14701 consensus_queue.cc:1035] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.934711 14617 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:41.955049 14713 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.956465 14617 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:41.956580 14470 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:41.958174 14618 raft_consensus.cc:1261] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:41.958329 14470 raft_consensus.cc:1261] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:41.978372 14470 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:41.978574 14701 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.980083 14713 consensus_queue.cc:1035] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.980918 14713 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.981855 14713 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.982841 14713 consensus_queue.cc:1035] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:42.152900 14509 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.219.131:39403, user_credentials={real_user=slave}} blocked reactor thread for 69031.5us
I20240501 20:09:42.168637 14468 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.172752 14543 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:42.169293 14669 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240501 20:09:42.173547 14710 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:42.176400 14738 consensus_queue.cc:1035] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240501 20:09:42.274381 14614 raft_consensus.cc:1261] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:09:42.335675 14750 log.cc:924] Time spent T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
W20240501 20:09:42.352154 14775 log.cc:924] Time spent T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20240501 20:09:42.355957 14737 log.cc:924] Time spent T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20240501 20:09:42.384155 14434 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.219.131:39403, user_credentials={real_user=slave}} blocked reactor thread for 75104.3us
I20240501 20:09:42.398428 14795 consensus_queue.cc:1035] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20240501 20:09:42.411779 14765 mvcc.cc:205] Tried to move back new op lower bound from 7022977770162520064 to 7022977768289529856. Current Snapshot: MvccSnapshot[applied={T|T < 7022977770155192320 or (T in {7022977770162520064,7022977770160070656})}]
W20240501 20:09:42.686549 14434 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.219.131:39403, user_credentials={real_user=slave}} blocked reactor thread for 120737us
I20240501 20:09:43.287000 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:43.299382 14811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:43.300451 14812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:43.301229 14814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:43.302431 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:43.302722 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:43.302780 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:43.302819 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594183302816 us; error 0 us; skew 500 ppm
I20240501 20:09:43.302959 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:43.304492 11118 webserver.cc:460] Webserver started at http://127.10.219.132:39859/ using document root <none> and password file <none>
I20240501 20:09:43.304744 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:43.304821 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:43.304934 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:43.305620 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-3-root/instance:
uuid: "12317e44612647c9bf2e942d55a89bfa"
format_stamp: "Formatted at 2024-05-01 20:09:43 on dist-test-slave-n6pz"
I20240501 20:09:43.308946 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 20:09:43.310911 14819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:43.311412 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 20:09:43.311547 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-3-root
uuid: "12317e44612647c9bf2e942d55a89bfa"
format_stamp: "Formatted at 2024-05-01 20:09:43 on dist-test-slave-n6pz"
I20240501 20:09:43.311702 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:43.331063 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:43.331761 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:43.332485 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:43.333771 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:43.333854 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:43.333953 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:43.334012 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:43.352097 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.132:44999
I20240501 20:09:43.352123 14881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.132:44999 every 8 connection(s)
I20240501 20:09:43.354408 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:43.358829 14886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:43.359685 14887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:43.361212 14882 heartbeater.cc:347] Connected to a master server at 127.10.219.190:41181
I20240501 20:09:43.361361 14882 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:43.361804 11118 server_base.cc:1034] running on GCE node
I20240501 20:09:43.361960 14882 heartbeater.cc:511] Master 127.10.219.190:41181 requested a full tablet report, sending...
W20240501 20:09:43.362529 14890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:43.362883 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:43.362944 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:43.362970 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594183362966 us; error 0 us; skew 500 ppm
I20240501 20:09:43.363003 14364 ts_manager.cc:194] Registered new tserver with Master: 12317e44612647c9bf2e942d55a89bfa (127.10.219.132:44999)
I20240501 20:09:43.363103 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:43.364197 14364 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54004
I20240501 20:09:43.364734 11118 webserver.cc:460] Webserver started at http://127.10.219.133:33805/ using document root <none> and password file <none>
I20240501 20:09:43.364981 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:43.365064 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:43.365202 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:43.365968 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-4-root/instance:
uuid: "62ad649017804e6faf6684ff1478d3b0"
format_stamp: "Formatted at 2024-05-01 20:09:43 on dist-test-slave-n6pz"
I20240501 20:09:43.368837 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:43.370672 14894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:43.371062 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 20:09:43.371160 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-4-root
uuid: "62ad649017804e6faf6684ff1478d3b0"
format_stamp: "Formatted at 2024-05-01 20:09:43 on dist-test-slave-n6pz"
I20240501 20:09:43.371295 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:43.383893 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:43.384588 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:43.385282 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:43.386451 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:43.386526 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:43.386638 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:43.386696 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:43.403178 14956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.133:46093 every 8 connection(s)
I20240501 20:09:43.403146 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.133:46093
I20240501 20:09:43.405310 11118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:43.409547 14960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:43.411311 11118 server_base.cc:1034] running on GCE node
W20240501 20:09:43.412508 14961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:43.413302 14963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:43.413847 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:09:43.413941 11118 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:09:43.413986 11118 hybrid_clock.cc:648] HybridClock initialized: now 1714594183413981 us; error 0 us; skew 500 ppm
I20240501 20:09:43.414175 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:43.414345 14957 heartbeater.cc:347] Connected to a master server at 127.10.219.190:41181
I20240501 20:09:43.414554 14957 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:43.415024 14957 heartbeater.cc:511] Master 127.10.219.190:41181 requested a full tablet report, sending...
I20240501 20:09:43.416388 11118 webserver.cc:460] Webserver started at http://127.10.219.134:35967/ using document root <none> and password file <none>
I20240501 20:09:43.416563 14364 ts_manager.cc:194] Registered new tserver with Master: 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133:46093)
I20240501 20:09:43.416694 11118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:43.416814 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:43.416958 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:43.417748 14364 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54016
I20240501 20:09:43.417878 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-5-root/instance:
uuid: "d1eb1382cb8f47b7aef9d61088b20974"
format_stamp: "Formatted at 2024-05-01 20:09:43 on dist-test-slave-n6pz"
I20240501 20:09:43.421123 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:09:43.423131 14968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:43.423590 11118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240501 20:09:43.423715 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-5-root
uuid: "d1eb1382cb8f47b7aef9d61088b20974"
format_stamp: "Formatted at 2024-05-01 20:09:43 on dist-test-slave-n6pz"
I20240501 20:09:43.423877 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:43.434125 11118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:43.434744 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:43.435348 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:43.436277 11118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:43.436342 11118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:43.436429 11118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:43.436467 11118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:43.453763 11118 rpc_server.cc:307] RPC server started. Bound to: 127.10.219.134:46221
I20240501 20:09:43.453819 15030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.219.134:46221 every 8 connection(s)
I20240501 20:09:43.462097 15031 heartbeater.cc:347] Connected to a master server at 127.10.219.190:41181
I20240501 20:09:43.462347 15031 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:43.462867 15031 heartbeater.cc:511] Master 127.10.219.190:41181 requested a full tablet report, sending...
I20240501 20:09:43.464205 14364 ts_manager.cc:194] Registered new tserver with Master: d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134:46221)
I20240501 20:09:43.465301 14364 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54026
I20240501 20:09:43.734514 14422 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:43.746512 14544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
new_leader_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:35692
I20240501 20:09:43.746819 14544 raft_consensus.cc:595] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Received request to transfer leadership to TS d66b9c8dab924baa8463d26f5f25b397
I20240501 20:09:43.748193 14544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
new_leader_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:35692
I20240501 20:09:43.748454 14544 raft_consensus.cc:595] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Received request to transfer leadership to TS d66b9c8dab924baa8463d26f5f25b397
I20240501 20:09:43.749483 14544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
new_leader_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:35692
I20240501 20:09:43.749660 14544 raft_consensus.cc:595] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Received request to transfer leadership to TS d66b9c8dab924baa8463d26f5f25b397
I20240501 20:09:43.750085 14422 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 20:09:43.750392 14422 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:43.754139 14465 consensus_queue.cc:237] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:43.756466 14615 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 20:09:43.756487 14544 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 20:09:43.757287 14696 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:43.758791 14696 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:43.761111 14738 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:43.761787 14544 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:43.767901 14364 catalog_manager.cc:5484] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index -1 to 26, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:43.769502 14615 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:43.775460 14544 consensus_queue.cc:237] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
W20240501 20:09:43.776391 14434 consensus_peers.cc:485] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c -> Peer 12317e44612647c9bf2e942d55a89bfa (127.10.219.132:44999): Couldn't send request to peer 12317e44612647c9bf2e942d55a89bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85e1a32af97d4a238310403e907092e0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:43.779389 14465 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 20:09:43.779389 14615 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 20:09:43.780331 14805 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:43.780838 14713 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:43.783448 14805 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:43.784144 14615 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:43.788028 14465 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:43.791075 14464 consensus_queue.cc:237] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:43.790019 14361 catalog_manager.cc:5484] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index -1 to 26, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:43.794236 14615 raft_consensus.cc:1261] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 20:09:43.794847 14738 consensus_queue.cc:1035] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:43.795276 14544 raft_consensus.cc:1261] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 20:09:43.796527 14738 consensus_queue.cc:1035] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:43.797960 14696 raft_consensus.cc:2932] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:43.799722 14615 raft_consensus.cc:2932] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:43.801518 14544 raft_consensus.cc:2932] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:43.807454 14464 consensus_queue.cc:237] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:43.807695 14361 catalog_manager.cc:5484] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index -1 to 26, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:43.812651 14544 raft_consensus.cc:1261] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 20:09:43.813329 14696 consensus_queue.cc:1035] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
W20240501 20:09:43.816274 14509 consensus_peers.cc:485] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 -> Peer d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134:46221): Couldn't send request to peer d1eb1382cb8f47b7aef9d61088b20974. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45bd961f0ac04a4a812a818a864c902f. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:43.818840 14433 consensus_peers.cc:485] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c -> Peer 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133:46093): Couldn't send request to peer 62ad649017804e6faf6684ff1478d3b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c5cf239f44e48e2a85f97db902642ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:43.822806 14738 raft_consensus.cc:2932] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:43.823787 14544 raft_consensus.cc:2932] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
W20240501 20:09:43.828263 14416 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20240501 20:09:43.828512 14364 catalog_manager.cc:5484] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: config changed from index -1 to 26, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:43.836531 14615 raft_consensus.cc:1261] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
W20240501 20:09:43.836681 14434 consensus_peers.cc:485] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c -> Peer d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134:46221): Couldn't send request to peer d1eb1382cb8f47b7aef9d61088b20974. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: baf26068e4c246baa73afccb4e93c847. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:43.837129 14738 consensus_queue.cc:1035] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
W20240501 20:09:43.837607 14416 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b4ddb456073f4c4aa53517b561c948af, TS d66b9c8dab924baa8463d26f5f25b397 -> TS 12317e44612647c9bf2e942d55a89bfa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
W20240501 20:09:43.837678 14416 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b4ddb456073f4c4aa53517b561c948af, TS d66b9c8dab924baa8463d26f5f25b397 -> TS 12317e44612647c9bf2e942d55a89bfa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:43.838917 14615 raft_consensus.cc:2932] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
W20240501 20:09:43.844580 14416 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet cd5f05007317454bb4e7bdefd81ccd90, TS 8d368a895e2a48c8a17b384c0739e47c -> TS 62ad649017804e6faf6684ff1478d3b0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
W20240501 20:09:43.844722 14416 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet cd5f05007317454bb4e7bdefd81ccd90, TS 8d368a895e2a48c8a17b384c0739e47c -> TS 62ad649017804e6faf6684ff1478d3b0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:43.995409 14712 raft_consensus.cc:979] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45: : Instructing follower d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:43.995656 14712 raft_consensus.cc:1067] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Signalling peer d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:43.996258 14615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:38276
I20240501 20:09:43.996438 14615 raft_consensus.cc:482] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:43.996544 14615 raft_consensus.cc:3037] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:43.999061 14615 raft_consensus.cc:504] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:43.999817 14615 leader_election.cc:288] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 2 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:44.000521 14465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:44.000591 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:44.001060 14544 raft_consensus.cc:3032] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:44.001066 14465 raft_consensus.cc:3037] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:44.001258 14544 raft_consensus.cc:729] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:44.001622 14544 consensus_queue.cc:260] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:44.002111 14544 raft_consensus.cc:3037] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:44.004904 14465 raft_consensus.cc:2454] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 2.
I20240501 20:09:44.005066 14544 raft_consensus.cc:2454] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 2.
I20240501 20:09:44.005676 14583 leader_election.cc:302] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:44.006130 14807 raft_consensus.cc:2781] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:44.006656 14807 raft_consensus.cc:686] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Becoming Leader. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:44.006984 14807 consensus_queue.cc:237] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:44.009728 14364 catalog_manager.cc:5484] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: term changed from 1 to 2, leader changed from 3da749f295434bceb714fb91c9598d45 (127.10.219.130) to d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131). New cstate: current_term: 2 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:44.020514 14713 raft_consensus.cc:979] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45: : Instructing follower d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:44.020722 14713 raft_consensus.cc:1067] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Signalling peer d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:44.021375 14617 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:38276
I20240501 20:09:44.021576 14617 raft_consensus.cc:482] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:44.021662 14617 raft_consensus.cc:3037] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:44.024585 14617 raft_consensus.cc:504] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:44.025672 14617 leader_election.cc:288] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 2 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:44.026046 14465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:44.026046 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:44.026273 14465 raft_consensus.cc:3037] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:44.026327 14543 raft_consensus.cc:3032] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:44.026427 14543 raft_consensus.cc:729] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:44.026681 14543 consensus_queue.cc:260] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:44.027040 14543 raft_consensus.cc:3037] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:44.029165 14543 raft_consensus.cc:2454] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 2.
I20240501 20:09:44.029166 14465 raft_consensus.cc:2454] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 2.
I20240501 20:09:44.029596 14585 leader_election.cc:302] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:44.029897 14807 raft_consensus.cc:2781] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:44.030014 14807 raft_consensus.cc:686] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Becoming Leader. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:44.030248 14807 consensus_queue.cc:237] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:44.033437 14364 catalog_manager.cc:5484] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: term changed from 1 to 2, leader changed from 3da749f295434bceb714fb91c9598d45 (127.10.219.130) to d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131). New cstate: current_term: 2 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:44.125059 14713 raft_consensus.cc:979] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45: : Instructing follower d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:44.125278 14713 raft_consensus.cc:1067] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Signalling peer d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:44.125965 14617 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:38276
I20240501 20:09:44.126171 14617 raft_consensus.cc:482] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:44.126283 14617 raft_consensus.cc:3037] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:44.128710 14617 raft_consensus.cc:504] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:44.129571 14617 leader_election.cc:288] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 2 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:44.129906 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:44.130086 14543 raft_consensus.cc:3032] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:44.130162 14543 raft_consensus.cc:729] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:44.130218 14465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:44.130460 14465 raft_consensus.cc:3037] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:44.130439 14543 consensus_queue.cc:260] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:44.130712 14543 raft_consensus.cc:3037] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:44.132584 14543 raft_consensus.cc:2454] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 2.
I20240501 20:09:44.132611 14465 raft_consensus.cc:2454] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 2.
I20240501 20:09:44.133028 14583 leader_election.cc:302] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:44.133304 14807 raft_consensus.cc:2781] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:44.133441 14807 raft_consensus.cc:686] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Becoming Leader. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:44.133955 14807 consensus_queue.cc:237] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:44.136776 14364 catalog_manager.cc:5484] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: term changed from 1 to 2, leader changed from 3da749f295434bceb714fb91c9598d45 (127.10.219.130) to d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131). New cstate: current_term: 2 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:44.169476 15050 ts_tablet_manager.cc:902] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0: Initiating tablet copy from peer 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765)
I20240501 20:09:44.171126 15050 tablet_copy_client.cc:308] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.129:33765
I20240501 20:09:44.177067 14480 tablet_copy_service.cc:140] P 8d368a895e2a48c8a17b384c0739e47c: Received BeginTabletCopySession request for tablet 7c5cf239f44e48e2a85f97db902642ac from peer 62ad649017804e6faf6684ff1478d3b0 ({username='slave'} at 127.0.0.1:44506)
I20240501 20:09:44.177296 14480 tablet_copy_service.cc:161] P 8d368a895e2a48c8a17b384c0739e47c: Beginning new tablet copy session on tablet 7c5cf239f44e48e2a85f97db902642ac from peer 62ad649017804e6faf6684ff1478d3b0 at {username='slave'} at 127.0.0.1:44506: session id = 62ad649017804e6faf6684ff1478d3b0-7c5cf239f44e48e2a85f97db902642ac
I20240501 20:09:44.179886 14480 tablet_copy_source_session.cc:205] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:44.181455 15050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5cf239f44e48e2a85f97db902642ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.187286 15050 tablet_copy_client.cc:783] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.187666 15050 tablet_copy_client.cc:647] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.189651 15054 tablet_copy_client.cc:1002] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:44.228365 15056 ts_tablet_manager.cc:902] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: Initiating tablet copy from peer 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765)
I20240501 20:09:44.230629 15056 tablet_copy_client.cc:308] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.129:33765
I20240501 20:09:44.237138 14480 tablet_copy_service.cc:140] P 8d368a895e2a48c8a17b384c0739e47c: Received BeginTabletCopySession request for tablet 85e1a32af97d4a238310403e907092e0 from peer 12317e44612647c9bf2e942d55a89bfa ({username='slave'} at 127.0.0.1:44522)
I20240501 20:09:44.237385 14480 tablet_copy_service.cc:161] P 8d368a895e2a48c8a17b384c0739e47c: Beginning new tablet copy session on tablet 85e1a32af97d4a238310403e907092e0 from peer 12317e44612647c9bf2e942d55a89bfa at {username='slave'} at 127.0.0.1:44522: session id = 12317e44612647c9bf2e942d55a89bfa-85e1a32af97d4a238310403e907092e0
I20240501 20:09:44.239959 14480 tablet_copy_source_session.cc:205] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:44.241307 15056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e1a32af97d4a238310403e907092e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.247761 15056 tablet_copy_client.cc:783] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.248090 15056 tablet_copy_client.cc:647] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.249686 15060 tablet_copy_client.cc:1002] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:44.264248 15062 ts_tablet_manager.cc:902] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: Initiating tablet copy from peer 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:44.265944 15062 tablet_copy_client.cc:308] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.130:40971
I20240501 20:09:44.273238 14555 tablet_copy_service.cc:140] P 3da749f295434bceb714fb91c9598d45: Received BeginTabletCopySession request for tablet 45bd961f0ac04a4a812a818a864c902f from peer d1eb1382cb8f47b7aef9d61088b20974 ({username='slave'} at 127.0.0.1:35720)
I20240501 20:09:44.273499 14555 tablet_copy_service.cc:161] P 3da749f295434bceb714fb91c9598d45: Beginning new tablet copy session on tablet 45bd961f0ac04a4a812a818a864c902f from peer d1eb1382cb8f47b7aef9d61088b20974 at {username='slave'} at 127.0.0.1:35720: session id = d1eb1382cb8f47b7aef9d61088b20974-45bd961f0ac04a4a812a818a864c902f
I20240501 20:09:44.276306 14555 tablet_copy_source_session.cc:205] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:44.278712 15062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bd961f0ac04a4a812a818a864c902f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.284657 15062 tablet_copy_client.cc:783] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.285213 15062 tablet_copy_client.cc:647] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.287307 15066 tablet_copy_client.cc:1002] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:44.365764 14882 heartbeater.cc:503] Master 127.10.219.190:41181 was elected leader, sending a full tablet report...
I20240501 20:09:44.419195 14957 heartbeater.cc:503] Master 127.10.219.190:41181 was elected leader, sending a full tablet report...
I20240501 20:09:44.421873 15070 ts_tablet_manager.cc:902] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974: Initiating tablet copy from peer 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765)
I20240501 20:09:44.423559 15070 tablet_copy_client.cc:308] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.129:33765
I20240501 20:09:44.429773 14480 tablet_copy_service.cc:140] P 8d368a895e2a48c8a17b384c0739e47c: Received BeginTabletCopySession request for tablet baf26068e4c246baa73afccb4e93c847 from peer d1eb1382cb8f47b7aef9d61088b20974 ({username='slave'} at 127.0.0.1:44536)
I20240501 20:09:44.429999 14480 tablet_copy_service.cc:161] P 8d368a895e2a48c8a17b384c0739e47c: Beginning new tablet copy session on tablet baf26068e4c246baa73afccb4e93c847 from peer d1eb1382cb8f47b7aef9d61088b20974 at {username='slave'} at 127.0.0.1:44536: session id = d1eb1382cb8f47b7aef9d61088b20974-baf26068e4c246baa73afccb4e93c847
I20240501 20:09:44.432720 14480 tablet_copy_source_session.cc:205] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:44.434442 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baf26068e4c246baa73afccb4e93c847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.440112 15070 tablet_copy_client.cc:783] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:44.440707 15070 tablet_copy_client.cc:647] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:44.442543 15074 tablet_copy_client.cc:1002] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:44.467234 15031 heartbeater.cc:503] Master 127.10.219.190:41181 was elected leader, sending a full tablet report...
I20240501 20:09:44.502511 14465 raft_consensus.cc:1261] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 20:09:44.503151 14807 consensus_queue.cc:1035] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 20:09:44.507958 14544 raft_consensus.cc:1261] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 20:09:44.508734 14807 consensus_queue.cc:1035] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 20:09:44.519233 14467 raft_consensus.cc:1261] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 20:09:44.520078 15049 consensus_queue.cc:1035] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 20:09:44.527447 14544 raft_consensus.cc:1261] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 20:09:44.528254 15049 consensus_queue.cc:1035] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 20:09:44.596923 14464 raft_consensus.cc:1261] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 20:09:44.597579 15049 consensus_queue.cc:1035] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:44.602077 14543 raft_consensus.cc:1261] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 20:09:44.603897 14807 consensus_queue.cc:1035] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:45.750751 14422 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:45.753587 14422 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:09:45.753846 14422 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:46.190109 15050 tablet_copy_client.cc:523] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:46.193421 15050 tablet_bootstrap.cc:492] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0: Bootstrap starting.
I20240501 20:09:46.236019 15050 tablet_bootstrap.cc:492] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:46.236491 15050 tablet_bootstrap.cc:492] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0: Bootstrap complete.
I20240501 20:09:46.236845 15050 ts_tablet_manager.cc:1367] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0: Time spent bootstrapping tablet: real 0.043s user 0.035s sys 0.009s
I20240501 20:09:46.237650 15050 raft_consensus.cc:348] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:46.237896 15050 raft_consensus.cc:729] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 62ad649017804e6faf6684ff1478d3b0, State: Initialized, Role: LEARNER
I20240501 20:09:46.238126 15050 consensus_queue.cc:260] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:46.239037 15050 ts_tablet_manager.cc:1398] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:09:46.239773 14480 tablet_copy_service.cc:342] P 8d368a895e2a48c8a17b384c0739e47c: Request end of tablet copy session 62ad649017804e6faf6684ff1478d3b0-7c5cf239f44e48e2a85f97db902642ac received from {username='slave'} at 127.0.0.1:44506
I20240501 20:09:46.239914 14480 tablet_copy_service.cc:433] P 8d368a895e2a48c8a17b384c0739e47c: ending tablet copy session 62ad649017804e6faf6684ff1478d3b0-7c5cf239f44e48e2a85f97db902642ac on tablet 7c5cf239f44e48e2a85f97db902642ac with peer 62ad649017804e6faf6684ff1478d3b0
I20240501 20:09:46.250136 15056 tablet_copy_client.cc:523] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:46.253834 15056 tablet_bootstrap.cc:492] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: Bootstrap starting.
I20240501 20:09:46.287755 15062 tablet_copy_client.cc:523] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:46.290661 15062 tablet_bootstrap.cc:492] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: Bootstrap starting.
I20240501 20:09:46.291085 15056 tablet_bootstrap.cc:492] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:46.291541 15056 tablet_bootstrap.cc:492] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: Bootstrap complete.
I20240501 20:09:46.291833 15056 ts_tablet_manager.cc:1367] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: Time spent bootstrapping tablet: real 0.038s user 0.032s sys 0.007s
I20240501 20:09:46.292605 15056 raft_consensus.cc:348] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:46.292872 15056 raft_consensus.cc:729] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12317e44612647c9bf2e942d55a89bfa, State: Initialized, Role: LEARNER
I20240501 20:09:46.293082 15056 consensus_queue.cc:260] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:46.300280 15056 ts_tablet_manager.cc:1398] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240501 20:09:46.301092 14480 tablet_copy_service.cc:342] P 8d368a895e2a48c8a17b384c0739e47c: Request end of tablet copy session 12317e44612647c9bf2e942d55a89bfa-85e1a32af97d4a238310403e907092e0 received from {username='slave'} at 127.0.0.1:44522
I20240501 20:09:46.301236 14480 tablet_copy_service.cc:433] P 8d368a895e2a48c8a17b384c0739e47c: ending tablet copy session 12317e44612647c9bf2e942d55a89bfa-85e1a32af97d4a238310403e907092e0 on tablet 85e1a32af97d4a238310403e907092e0 with peer 12317e44612647c9bf2e942d55a89bfa
I20240501 20:09:46.311332 14932 raft_consensus.cc:1203] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240501 20:09:46.332470 15062 tablet_bootstrap.cc:492] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:46.332952 15062 tablet_bootstrap.cc:492] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: Bootstrap complete.
I20240501 20:09:46.333242 15062 ts_tablet_manager.cc:1367] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: Time spent bootstrapping tablet: real 0.043s user 0.038s sys 0.000s
I20240501 20:09:46.333963 15062 raft_consensus.cc:348] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:46.334208 15062 raft_consensus.cc:729] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1eb1382cb8f47b7aef9d61088b20974, State: Initialized, Role: LEARNER
I20240501 20:09:46.334436 15062 consensus_queue.cc:260] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:46.335268 15062 ts_tablet_manager.cc:1398] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 20:09:46.335920 14555 tablet_copy_service.cc:342] P 3da749f295434bceb714fb91c9598d45: Request end of tablet copy session d1eb1382cb8f47b7aef9d61088b20974-45bd961f0ac04a4a812a818a864c902f received from {username='slave'} at 127.0.0.1:35720
I20240501 20:09:46.336015 14555 tablet_copy_service.cc:433] P 3da749f295434bceb714fb91c9598d45: ending tablet copy session d1eb1382cb8f47b7aef9d61088b20974-45bd961f0ac04a4a812a818a864c902f on tablet 45bd961f0ac04a4a812a818a864c902f with peer d1eb1382cb8f47b7aef9d61088b20974
I20240501 20:09:46.443042 15070 tablet_copy_client.cc:523] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:46.446784 15070 tablet_bootstrap.cc:492] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974: Bootstrap starting.
I20240501 20:09:46.485121 15070 tablet_bootstrap.cc:492] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:46.485574 15070 tablet_bootstrap.cc:492] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974: Bootstrap complete.
I20240501 20:09:46.485862 15070 ts_tablet_manager.cc:1367] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974: Time spent bootstrapping tablet: real 0.039s user 0.028s sys 0.007s
I20240501 20:09:46.486667 15070 raft_consensus.cc:348] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:46.486935 15070 raft_consensus.cc:729] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1eb1382cb8f47b7aef9d61088b20974, State: Initialized, Role: LEARNER
I20240501 20:09:46.487128 15070 consensus_queue.cc:260] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:46.487707 15070 ts_tablet_manager.cc:1398] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:09:46.488333 14480 tablet_copy_service.cc:342] P 8d368a895e2a48c8a17b384c0739e47c: Request end of tablet copy session d1eb1382cb8f47b7aef9d61088b20974-baf26068e4c246baa73afccb4e93c847 received from {username='slave'} at 127.0.0.1:44536
I20240501 20:09:46.488463 14480 tablet_copy_service.cc:433] P 8d368a895e2a48c8a17b384c0739e47c: ending tablet copy session d1eb1382cb8f47b7aef9d61088b20974-baf26068e4c246baa73afccb4e93c847 on tablet baf26068e4c246baa73afccb4e93c847 with peer d1eb1382cb8f47b7aef9d61088b20974
I20240501 20:09:46.588400 14857 raft_consensus.cc:1203] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240501 20:09:46.694265 15006 raft_consensus.cc:1203] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240501 20:09:46.780189 15096 raft_consensus.cc:1050] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c: attempting to promote NON_VOTER 62ad649017804e6faf6684ff1478d3b0 to VOTER
I20240501 20:09:46.781159 15096 consensus_queue.cc:237] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } }
I20240501 20:09:46.783799 14932 raft_consensus.cc:1261] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [term 1 LEARNER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:46.783802 14617 raft_consensus.cc:1261] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:46.783802 14544 raft_consensus.cc:1261] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:46.784484 15098 consensus_queue.cc:1035] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:46.784929 15096 consensus_queue.cc:1035] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:46.785390 15069 consensus_queue.cc:1035] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:46.789927 15069 raft_consensus.cc:2932] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:46.790627 14932 raft_consensus.cc:2932] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:46.795337 14364 catalog_manager.cc:5484] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 reported cstate change: config changed from index 26 to 27, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:46.795930 14617 raft_consensus.cc:2932] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:46.797528 14544 raft_consensus.cc:2932] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:46.816049 14464 consensus_queue.cc:237] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } }
I20240501 20:09:46.818411 14932 raft_consensus.cc:1261] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 20:09:46.818502 14617 raft_consensus.cc:1261] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 20:09:46.819015 15096 consensus_queue.cc:1035] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:46.819357 15098 consensus_queue.cc:1035] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:46.821911 15069 raft_consensus.cc:2932] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:46.822527 14617 raft_consensus.cc:2932] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:46.825623 14353 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c5cf239f44e48e2a85f97db902642ac with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:46.827354 14932 raft_consensus.cc:2932] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:46.827644 14359 catalog_manager.cc:5484] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: config changed from index 27 to 28, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:46.829706 15006 raft_consensus.cc:1203] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240501 20:09:46.834424 14533 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c5cf239f44e48e2a85f97db902642ac with delete_type TABLET_DATA_TOMBSTONED (TS 3da749f295434bceb714fb91c9598d45 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:35656
I20240501 20:09:46.835283 15106 tablet_replica.cc:330] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45: stopping tablet replica
I20240501 20:09:46.835580 15106 raft_consensus.cc:2229] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:46.835865 15106 raft_consensus.cc:2258] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:46.837013 15106 ts_tablet_manager.cc:1875] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:46.843034 15106 ts_tablet_manager.cc:1888] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240501 20:09:46.843349 15106 log.cc:1196] T 7c5cf239f44e48e2a85f97db902642ac P 3da749f295434bceb714fb91c9598d45: Deleting WAL directory at /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-1-root/wals/7c5cf239f44e48e2a85f97db902642ac
I20240501 20:09:46.844295 14355 catalog_manager.cc:4830] TS 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971): tablet 7c5cf239f44e48e2a85f97db902642ac (table test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]) successfully deleted
I20240501 20:09:46.978168 15096 raft_consensus.cc:1050] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c: attempting to promote NON_VOTER 12317e44612647c9bf2e942d55a89bfa to VOTER
I20240501 20:09:46.979233 15096 consensus_queue.cc:237] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:46.981657 14857 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 1 LEARNER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:46.981657 14544 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:46.981731 14617 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:46.982374 15096 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:46.982736 15095 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:46.983006 15069 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:46.986748 15096 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:46.987468 14617 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:46.993510 14361 catalog_manager.cc:5484] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: config changed from index 26 to 27, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:46.994119 14544 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:46.999925 14857 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:47.003865 14464 consensus_queue.cc:237] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:47.006114 14857 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 20:09:47.006388 14544 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 20:09:47.006896 15069 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:47.007158 15096 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:47.009706 15098 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:47.013428 14353 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 85e1a32af97d4a238310403e907092e0 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:47.013592 14543 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:47.013826 14857 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:47.014901 14361 catalog_manager.cc:5484] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: config changed from index 27 to 28, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:47.019575 14607 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e1a32af97d4a238310403e907092e0 with delete_type TABLET_DATA_TOMBSTONED (TS d66b9c8dab924baa8463d26f5f25b397 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:38258
I20240501 20:09:47.020424 15111 tablet_replica.cc:330] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397: stopping tablet replica
I20240501 20:09:47.020733 15111 raft_consensus.cc:2229] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:47.021210 15111 raft_consensus.cc:2258] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:47.022241 15111 ts_tablet_manager.cc:1875] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:47.028520 15111 ts_tablet_manager.cc:1888] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240501 20:09:47.028712 15111 log.cc:1196] T 85e1a32af97d4a238310403e907092e0 P d66b9c8dab924baa8463d26f5f25b397: Deleting WAL directory at /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-2-root/wals/85e1a32af97d4a238310403e907092e0
I20240501 20:09:47.029536 14353 catalog_manager.cc:4830] TS d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403): tablet 85e1a32af97d4a238310403e907092e0 (table test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]) successfully deleted
I20240501 20:09:47.112716 15112 raft_consensus.cc:1050] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45: attempting to promote NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 to VOTER
I20240501 20:09:47.113569 15112 consensus_queue.cc:237] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:47.116816 14467 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:47.117314 14617 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:47.117405 15115 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:47.117851 15006 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 1 LEARNER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:47.118075 15112 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:47.118619 15112 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:47.121560 15087 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.122848 14617 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.125919 15006 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.126628 14467 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.127187 14368 catalog_manager.cc:5484] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:47.132761 14543 consensus_queue.cc:237] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:47.135143 14617 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 20:09:47.135143 15006 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 20:09:47.135856 15112 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:47.136198 15087 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:47.138669 15112 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.139349 14617 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.140115 15006 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.143473 14355 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 45bd961f0ac04a4a812a818a864c902f with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:47.144548 14359 catalog_manager.cc:5484] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 reported cstate change: config changed from index 27 to 28, VOTER 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) evicted. New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.148612 14453 tablet_service.cc:1514] Processing DeleteTablet for tablet 45bd961f0ac04a4a812a818a864c902f with delete_type TABLET_DATA_TOMBSTONED (TS 8d368a895e2a48c8a17b384c0739e47c not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:44444
I20240501 20:09:47.149600 15119 tablet_replica.cc:330] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c: stopping tablet replica
I20240501 20:09:47.149832 15119 raft_consensus.cc:2229] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:47.150218 15119 raft_consensus.cc:2258] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:47.151293 15119 ts_tablet_manager.cc:1875] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:47.157707 15119 ts_tablet_manager.cc:1888] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240501 20:09:47.157847 15119 log.cc:1196] T 45bd961f0ac04a4a812a818a864c902f P 8d368a895e2a48c8a17b384c0739e47c: Deleting WAL directory at /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-0-root/wals/45bd961f0ac04a4a812a818a864c902f
I20240501 20:09:47.158658 14353 catalog_manager.cc:4830] TS 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765): tablet 45bd961f0ac04a4a812a818a864c902f (table test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]) successfully deleted
I20240501 20:09:47.237669 15095 raft_consensus.cc:1050] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c: attempting to promote NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 to VOTER
I20240501 20:09:47.239071 15095 consensus_queue.cc:237] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:47.243551 15006 raft_consensus.cc:1261] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974 [term 1 LEARNER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:47.244400 14544 raft_consensus.cc:1261] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:47.244549 14617 raft_consensus.cc:1261] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:47.244650 15096 consensus_queue.cc:1035] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:47.245678 15069 consensus_queue.cc:1035] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:47.246824 15098 consensus_queue.cc:1035] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:47.249244 15098 raft_consensus.cc:2932] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.250044 14617 raft_consensus.cc:2932] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.252784 14544 raft_consensus.cc:2932] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.253940 15006 raft_consensus.cc:2932] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.254451 14361 catalog_manager.cc:5484] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:47.261360 14464 consensus_queue.cc:237] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:47.264364 14617 raft_consensus.cc:1261] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 20:09:47.265017 15005 raft_consensus.cc:1261] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974 [term 1 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 20:09:47.266428 15098 consensus_queue.cc:1035] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:47.267182 15098 consensus_queue.cc:1035] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:47.270020 15096 raft_consensus.cc:2932] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.270900 14617 raft_consensus.cc:2932] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.271577 15006 raft_consensus.cc:2932] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:47.275226 14353 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet baf26068e4c246baa73afccb4e93c847 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:47.276609 14361 catalog_manager.cc:5484] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: config changed from index 27 to 28, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New cstate: current_term: 1 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:47.281296 14533 tablet_service.cc:1514] Processing DeleteTablet for tablet baf26068e4c246baa73afccb4e93c847 with delete_type TABLET_DATA_TOMBSTONED (TS 3da749f295434bceb714fb91c9598d45 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:35656
I20240501 20:09:47.281702 15106 tablet_replica.cc:330] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45: stopping tablet replica
I20240501 20:09:47.281883 15106 raft_consensus.cc:2229] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:47.282146 15106 raft_consensus.cc:2258] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:47.283241 15106 ts_tablet_manager.cc:1875] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:47.288472 15106 ts_tablet_manager.cc:1888] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240501 20:09:47.288640 15106 log.cc:1196] T baf26068e4c246baa73afccb4e93c847 P 3da749f295434bceb714fb91c9598d45: Deleting WAL directory at /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-1-root/wals/baf26068e4c246baa73afccb4e93c847
I20240501 20:09:47.289497 14355 catalog_manager.cc:4830] TS 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971): tablet baf26068e4c246baa73afccb4e93c847 (table test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]) successfully deleted
I20240501 20:09:47.754230 14422 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:47.757618 14543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45bd961f0ac04a4a812a818a864c902f"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
new_leader_uuid: "d1eb1382cb8f47b7aef9d61088b20974"
from {username='slave'} at 127.0.0.1:35692
I20240501 20:09:47.757920 14543 raft_consensus.cc:595] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Received request to transfer leadership to TS d1eb1382cb8f47b7aef9d61088b20974
I20240501 20:09:47.771699 14464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b90dffdaa47425883abd651fe391006"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
new_leader_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:55242
I20240501 20:09:47.771981 14464 raft_consensus.cc:595] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Received request to transfer leadership to TS 3da749f295434bceb714fb91c9598d45
I20240501 20:09:47.773202 14464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7c5cf239f44e48e2a85f97db902642ac"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
new_leader_uuid: "62ad649017804e6faf6684ff1478d3b0"
from {username='slave'} at 127.0.0.1:55242
I20240501 20:09:47.773429 14464 raft_consensus.cc:595] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Received request to transfer leadership to TS 62ad649017804e6faf6684ff1478d3b0
I20240501 20:09:47.780231 14617 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
mode: GRACEFUL
new_leader_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:34480
I20240501 20:09:47.780485 14617 raft_consensus.cc:595] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Received request to transfer leadership to TS 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:47.781723 14543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
new_leader_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:35692
I20240501 20:09:47.781912 14543 raft_consensus.cc:595] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Received request to transfer leadership to TS 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:47.782341 14422 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240501 20:09:47.782543 14422 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:47.981608 15095 raft_consensus.cc:979] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c: : Instructing follower 62ad649017804e6faf6684ff1478d3b0 to start an election
I20240501 20:09:47.981760 15095 raft_consensus.cc:1067] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Signalling peer 62ad649017804e6faf6684ff1478d3b0 to start an election
I20240501 20:09:47.982504 14932 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c5cf239f44e48e2a85f97db902642ac"
dest_uuid: "62ad649017804e6faf6684ff1478d3b0"
from {username='slave'} at 127.0.0.1:43082
I20240501 20:09:47.982715 14932 raft_consensus.cc:482] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:47.982792 14932 raft_consensus.cc:3037] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:47.984872 14932 raft_consensus.cc:504] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } }
I20240501 20:09:47.985750 14932 leader_election.cc:288] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:47.986385 14464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5cf239f44e48e2a85f97db902642ac" candidate_uuid: "62ad649017804e6faf6684ff1478d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:47.986635 14464 raft_consensus.cc:3032] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:47.986716 14464 raft_consensus.cc:729] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:47.987004 14464 consensus_queue.cc:260] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } }
I20240501 20:09:47.987417 14464 raft_consensus.cc:3037] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:47.989645 14464 raft_consensus.cc:2454] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62ad649017804e6faf6684ff1478d3b0 in term 2.
I20240501 20:09:47.990121 14896 leader_election.cc:302] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ad649017804e6faf6684ff1478d3b0, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:47.990765 15130 raft_consensus.cc:2781] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:47.991189 15130 raft_consensus.cc:686] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [term 2 LEADER]: Becoming Leader. State: Replica: 62ad649017804e6faf6684ff1478d3b0, State: Running, Role: LEADER
I20240501 20:09:47.991490 15130 consensus_queue.cc:237] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } }
I20240501 20:09:47.992527 14617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5cf239f44e48e2a85f97db902642ac" candidate_uuid: "62ad649017804e6faf6684ff1478d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:47.992753 14617 raft_consensus.cc:3037] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:47.995157 14617 raft_consensus.cc:2454] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62ad649017804e6faf6684ff1478d3b0 in term 2.
I20240501 20:09:47.995112 14359 catalog_manager.cc:5484] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 reported cstate change: term changed from 1 to 2, leader changed from 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) to 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133). New cstate: current_term: 2 leader_uuid: "62ad649017804e6faf6684ff1478d3b0" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:48.221087 15136 raft_consensus.cc:979] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45: : Instructing follower d1eb1382cb8f47b7aef9d61088b20974 to start an election
I20240501 20:09:48.221268 15136 raft_consensus.cc:1067] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Signalling peer d1eb1382cb8f47b7aef9d61088b20974 to start an election
I20240501 20:09:48.221875 15006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "45bd961f0ac04a4a812a818a864c902f"
dest_uuid: "d1eb1382cb8f47b7aef9d61088b20974"
from {username='slave'} at 127.0.0.1:53062
I20240501 20:09:48.222067 15006 raft_consensus.cc:482] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:48.222174 15006 raft_consensus.cc:3037] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.224426 15006 raft_consensus.cc:504] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:48.225473 15006 leader_election.cc:288] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [CANDIDATE]: Term 2 election: Requested vote from peers d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:48.226106 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bd961f0ac04a4a812a818a864c902f" candidate_uuid: "d1eb1382cb8f47b7aef9d61088b20974" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:48.226286 14543 raft_consensus.cc:3032] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:48.226408 14543 raft_consensus.cc:729] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:48.226716 14543 consensus_queue.cc:260] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:48.227131 14543 raft_consensus.cc:3037] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.229807 14543 raft_consensus.cc:2454] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1eb1382cb8f47b7aef9d61088b20974 in term 2.
I20240501 20:09:48.230371 14972 leader_election.cc:302] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d1eb1382cb8f47b7aef9d61088b20974; no voters:
I20240501 20:09:48.232568 14617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bd961f0ac04a4a812a818a864c902f" candidate_uuid: "d1eb1382cb8f47b7aef9d61088b20974" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:48.232869 14617 raft_consensus.cc:3037] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.234220 15087 raft_consensus.cc:979] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45: : Instructing follower 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:48.239084 14617 raft_consensus.cc:2454] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1eb1382cb8f47b7aef9d61088b20974 in term 2.
I20240501 20:09:48.240154 15087 raft_consensus.cc:1067] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Signalling peer 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:48.241195 14464 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:44456
I20240501 20:09:48.241508 14464 raft_consensus.cc:482] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:48.241627 14464 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.245003 15138 raft_consensus.cc:2781] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:48.245493 14464 raft_consensus.cc:504] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:48.245688 15138 raft_consensus.cc:686] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEADER]: Becoming Leader. State: Replica: d1eb1382cb8f47b7aef9d61088b20974, State: Running, Role: LEADER
I20240501 20:09:48.246071 15138 consensus_queue.cc:237] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:48.246740 14464 leader_election.cc:288] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 2 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:48.247351 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:48.247560 14543 raft_consensus.cc:3032] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:48.247633 14543 raft_consensus.cc:729] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:48.247879 14543 consensus_queue.cc:260] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:48.248189 14543 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.249287 14359 catalog_manager.cc:5484] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 reported cstate change: term changed from 1 to 2, leader changed from 3da749f295434bceb714fb91c9598d45 (127.10.219.130) to d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134). New cstate: current_term: 2 leader_uuid: "d1eb1382cb8f47b7aef9d61088b20974" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:48.250438 14543 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 2.
I20240501 20:09:48.250939 14436 leader_election.cc:302] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:48.251718 15095 raft_consensus.cc:2781] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:48.251878 15095 raft_consensus.cc:686] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Becoming Leader. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:48.252185 15095 consensus_queue.cc:237] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:48.247341 14617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:48.252516 14617 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.255676 14617 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 2.
I20240501 20:09:48.255913 14359 catalog_manager.cc:5484] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: term changed from 1 to 2, leader changed from 3da749f295434bceb714fb91c9598d45 (127.10.219.130) to 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129). New cstate: current_term: 2 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:48.283291 14543 consensus_queue.cc:237] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:48.284202 15143 raft_consensus.cc:979] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397: : Instructing follower 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:48.284374 15143 raft_consensus.cc:1067] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Signalling peer 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:48.285079 14464 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:44462
I20240501 20:09:48.285321 14464 raft_consensus.cc:482] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:48.285429 14464 raft_consensus.cc:3037] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:48.287961 14464 raft_consensus.cc:504] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:48.288635 14464 leader_election.cc:288] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 3 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:48.289098 14467 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 20:09:48.289163 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:48.289433 14543 raft_consensus.cc:3037] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:48.289921 14615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:48.290050 14617 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 20:09:48.290089 14615 raft_consensus.cc:3032] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:09:48.290145 14615 raft_consensus.cc:729] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:48.290503 15136 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:48.290473 14615 consensus_queue.cc:260] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:48.291039 14615 raft_consensus.cc:3037] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:48.291435 15087 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:48.294538 14615 raft_consensus.cc:2454] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 3.
I20240501 20:09:48.295229 14434 leader_election.cc:302] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:48.295677 15095 raft_consensus.cc:2781] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:09:48.295832 15095 raft_consensus.cc:686] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 3 LEADER]: Becoming Leader. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:48.295878 14543 raft_consensus.cc:2454] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 3.
I20240501 20:09:48.296221 15095 consensus_queue.cc:237] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:48.299898 15087 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:48.303146 14615 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:48.305680 14467 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:48.307498 14359 catalog_manager.cc:5484] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index -1 to 26, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:48.315466 14359 catalog_manager.cc:5484] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: term changed from 2 to 3, leader changed from d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) to 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129). New cstate: current_term: 3 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:48.316484 14508 consensus_peers.cc:485] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 -> Peer 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133:46093): Couldn't send request to peer 62ad649017804e6faf6684ff1478d3b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9784e2a5d4324c63b42d41fe0fde222b. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:48.318436 14416 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica d66b9c8dab924baa8463d26f5f25b397 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } }
W20240501 20:09:48.322181 14416 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 8bf8ec6baf414c598a80753fe882d9a0, TS 8d368a895e2a48c8a17b384c0739e47c -> TS 12317e44612647c9bf2e942d55a89bfa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
W20240501 20:09:48.322377 14416 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 8bf8ec6baf414c598a80753fe882d9a0, TS 8d368a895e2a48c8a17b384c0739e47c -> TS 12317e44612647c9bf2e942d55a89bfa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:48.322669 15128 raft_consensus.cc:979] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c: : Instructing follower 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:48.322804 15128 raft_consensus.cc:1067] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Signalling peer 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:48.323450 14544 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b90dffdaa47425883abd651fe391006"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:35668
I20240501 20:09:48.323617 14544 raft_consensus.cc:482] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:48.323716 14544 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.326453 14544 raft_consensus.cc:504] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:48.327250 14544 leader_election.cc:288] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 2 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:48.327762 14615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:48.327960 14615 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:09:48.329619 14416 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b4ddb456073f4c4aa53517b561c948af, TS 8d368a895e2a48c8a17b384c0739e47c -> TS d1eb1382cb8f47b7aef9d61088b20974 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
W20240501 20:09:48.329717 14416 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b4ddb456073f4c4aa53517b561c948af, TS 8d368a895e2a48c8a17b384c0739e47c -> TS d1eb1382cb8f47b7aef9d61088b20974 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:48.330879 14615 raft_consensus.cc:2454] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 2.
I20240501 20:09:48.331236 14467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:48.331331 14509 leader_election.cc:302] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:48.331441 14467 raft_consensus.cc:3032] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:48.331507 14467 raft_consensus.cc:729] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:48.331811 15087 raft_consensus.cc:2781] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:48.331956 15087 raft_consensus.cc:686] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 2 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:48.331794 14467 consensus_queue.cc:260] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:48.332177 15087 consensus_queue.cc:237] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:48.332451 14467 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:48.335352 14467 raft_consensus.cc:2454] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 2.
I20240501 20:09:48.335402 14359 catalog_manager.cc:5484] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 1 to 2, leader changed from 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 2 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:48.336474 14416 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7b90dffdaa47425883abd651fe391006, TS 3da749f295434bceb714fb91c9598d45 -> TS 12317e44612647c9bf2e942d55a89bfa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
W20240501 20:09:48.336580 14416 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7b90dffdaa47425883abd651fe391006, TS 3da749f295434bceb714fb91c9598d45 -> TS 12317e44612647c9bf2e942d55a89bfa move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
W20240501 20:09:48.339915 14416 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet baf26068e4c246baa73afccb4e93c847, TS d66b9c8dab924baa8463d26f5f25b397 -> TS 62ad649017804e6faf6684ff1478d3b0 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
W20240501 20:09:48.340049 14416 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet baf26068e4c246baa73afccb4e93c847, TS d66b9c8dab924baa8463d26f5f25b397 -> TS 62ad649017804e6faf6684ff1478d3b0 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
W20240501 20:09:48.342948 14416 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet cd5f05007317454bb4e7bdefd81ccd90, TS 3da749f295434bceb714fb91c9598d45 -> TS d1eb1382cb8f47b7aef9d61088b20974 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
W20240501 20:09:48.343065 14416 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet cd5f05007317454bb4e7bdefd81ccd90, TS 3da749f295434bceb714fb91c9598d45 -> TS d1eb1382cb8f47b7aef9d61088b20974 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:48.417546 14464 raft_consensus.cc:1261] T 7c5cf239f44e48e2a85f97db902642ac P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Refusing update from remote peer 62ad649017804e6faf6684ff1478d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 20:09:48.418169 15130 consensus_queue.cc:1035] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:48.423972 14615 raft_consensus.cc:1261] T 7c5cf239f44e48e2a85f97db902642ac P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Refusing update from remote peer 62ad649017804e6faf6684ff1478d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 20:09:48.424752 15130 consensus_queue.cc:1035] T 7c5cf239f44e48e2a85f97db902642ac P 62ad649017804e6faf6684ff1478d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:48.654078 14544 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 20:09:48.655025 15095 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 20:09:48.656001 14615 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Refusing update from remote peer d1eb1382cb8f47b7aef9d61088b20974: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 20:09:48.656540 15138 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:48.659039 14617 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 20:09:48.659646 15128 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 20:09:48.674616 14544 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer d1eb1382cb8f47b7aef9d61088b20974: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 20:09:48.675289 15138 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:48.726589 14544 raft_consensus.cc:1261] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240501 20:09:48.727216 15128 consensus_queue.cc:1035] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:48.731740 14615 raft_consensus.cc:1261] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240501 20:09:48.732420 15159 consensus_queue.cc:1035] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:48.818185 15167 ts_tablet_manager.cc:902] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0: Initiating tablet copy from peer 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:48.819839 15167 tablet_copy_client.cc:308] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.130:40971
I20240501 20:09:48.827164 14555 tablet_copy_service.cc:140] P 3da749f295434bceb714fb91c9598d45: Received BeginTabletCopySession request for tablet 9784e2a5d4324c63b42d41fe0fde222b from peer 62ad649017804e6faf6684ff1478d3b0 ({username='slave'} at 127.0.0.1:47562)
I20240501 20:09:48.827345 14555 tablet_copy_service.cc:161] P 3da749f295434bceb714fb91c9598d45: Beginning new tablet copy session on tablet 9784e2a5d4324c63b42d41fe0fde222b from peer 62ad649017804e6faf6684ff1478d3b0 at {username='slave'} at 127.0.0.1:47562: session id = 62ad649017804e6faf6684ff1478d3b0-9784e2a5d4324c63b42d41fe0fde222b
I20240501 20:09:48.829950 14555 tablet_copy_source_session.cc:205] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:48.831359 15167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9784e2a5d4324c63b42d41fe0fde222b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.837044 15167 tablet_copy_client.cc:783] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:48.837369 15167 tablet_copy_client.cc:647] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:48.839200 15171 tablet_copy_client.cc:1002] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:48.847895 14615 raft_consensus.cc:1261] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 20:09:48.848462 15087 consensus_queue.cc:1035] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 20:09:48.852901 14464 raft_consensus.cc:1261] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 20:09:48.853533 15176 consensus_queue.cc:1035] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 20:09:49.782860 14422 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:49.786219 14544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b90dffdaa47425883abd651fe391006"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
new_leader_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:35692
I20240501 20:09:49.786515 14544 raft_consensus.cc:595] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 2 LEADER]: Received request to transfer leadership to TS d66b9c8dab924baa8463d26f5f25b397
I20240501 20:09:49.787631 14464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "85e1a32af97d4a238310403e907092e0"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
new_leader_uuid: "12317e44612647c9bf2e942d55a89bfa"
from {username='slave'} at 127.0.0.1:55242
I20240501 20:09:49.787820 14464 raft_consensus.cc:595] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Received request to transfer leadership to TS 12317e44612647c9bf2e942d55a89bfa
I20240501 20:09:49.788908 14615 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
mode: GRACEFUL
new_leader_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:34480
I20240501 20:09:49.789108 14615 raft_consensus.cc:595] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Received request to transfer leadership to TS 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:49.790153 14464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
new_leader_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:55242
I20240501 20:09:49.790343 14464 raft_consensus.cc:595] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Received request to transfer leadership to TS 3da749f295434bceb714fb91c9598d45
I20240501 20:09:49.790756 14422 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240501 20:09:49.791005 14422 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:49.896466 15182 raft_consensus.cc:979] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c: : Instructing follower 12317e44612647c9bf2e942d55a89bfa to start an election
I20240501 20:09:49.896689 15128 raft_consensus.cc:1067] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Signalling peer 12317e44612647c9bf2e942d55a89bfa to start an election
I20240501 20:09:49.897387 14857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "85e1a32af97d4a238310403e907092e0"
dest_uuid: "12317e44612647c9bf2e942d55a89bfa"
from {username='slave'} at 127.0.0.1:59378
I20240501 20:09:49.897647 14857 raft_consensus.cc:482] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:49.897794 14857 raft_consensus.cc:3037] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:49.900720 14857 raft_consensus.cc:504] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:49.901647 14857 leader_election.cc:288] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [CANDIDATE]: Term 2 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:49.902071 14464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e1a32af97d4a238310403e907092e0" candidate_uuid: "12317e44612647c9bf2e942d55a89bfa" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:49.902398 14464 raft_consensus.cc:3032] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:49.902534 14464 raft_consensus.cc:729] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:49.902887 14464 consensus_queue.cc:260] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:49.903280 14464 raft_consensus.cc:3037] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:49.906164 14464 raft_consensus.cc:2454] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12317e44612647c9bf2e942d55a89bfa in term 2.
I20240501 20:09:49.906695 14821 leader_election.cc:302] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12317e44612647c9bf2e942d55a89bfa, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:49.912572 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e1a32af97d4a238310403e907092e0" candidate_uuid: "12317e44612647c9bf2e942d55a89bfa" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:49.912918 14543 raft_consensus.cc:3037] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:49.913015 15186 raft_consensus.cc:2781] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:49.913496 15186 raft_consensus.cc:686] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 2 LEADER]: Becoming Leader. State: Replica: 12317e44612647c9bf2e942d55a89bfa, State: Running, Role: LEADER
I20240501 20:09:49.914090 15186 consensus_queue.cc:237] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:49.915803 14543 raft_consensus.cc:2454] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12317e44612647c9bf2e942d55a89bfa in term 2.
I20240501 20:09:49.917189 14368 catalog_manager.cc:5484] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa reported cstate change: term changed from 1 to 2, leader changed from 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) to 12317e44612647c9bf2e942d55a89bfa (127.10.219.132). New cstate: current_term: 2 leader_uuid: "12317e44612647c9bf2e942d55a89bfa" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:50.145891 15193 raft_consensus.cc:979] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397: : Instructing follower 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:50.146119 15193 raft_consensus.cc:1067] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Signalling peer 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:50.146914 14464 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:44462
I20240501 20:09:50.147135 14464 raft_consensus.cc:482] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:50.147253 14464 raft_consensus.cc:3037] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:50.149763 14464 raft_consensus.cc:504] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:50.150840 14464 leader_election.cc:288] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 3 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:50.151021 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:50.151283 14543 raft_consensus.cc:3037] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:50.151054 14615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c570e28c114771a13fcac4fa94af43" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:50.151496 14615 raft_consensus.cc:3032] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:09:50.151602 14615 raft_consensus.cc:729] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:50.151952 14615 consensus_queue.cc:260] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:50.152380 14615 raft_consensus.cc:3037] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:50.155388 14615 raft_consensus.cc:2454] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 3.
I20240501 20:09:50.155898 14434 leader_election.cc:302] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:50.156052 14543 raft_consensus.cc:2454] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 3.
I20240501 20:09:50.156284 15182 raft_consensus.cc:2781] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:09:50.156471 15182 raft_consensus.cc:686] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 3 LEADER]: Becoming Leader. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:50.156759 15182 consensus_queue.cc:237] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:50.160909 14368 catalog_manager.cc:5484] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: term changed from 2 to 3, leader changed from d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) to 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129). New cstate: current_term: 3 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:50.182523 15175 raft_consensus.cc:979] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45: : Instructing follower d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:50.182737 15176 raft_consensus.cc:1067] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 2 LEADER]: Signalling peer d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:50.183532 14615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b90dffdaa47425883abd651fe391006"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:38276
I20240501 20:09:50.183794 14615 raft_consensus.cc:482] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:50.183931 14615 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:50.186288 14615 raft_consensus.cc:504] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:50.186997 14615 leader_election.cc:288] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 3 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:50.187404 14464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:50.187569 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:50.187644 14464 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:50.187728 14543 raft_consensus.cc:3032] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:09:50.187795 14543 raft_consensus.cc:729] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:50.188067 14543 consensus_queue.cc:260] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:50.188643 14543 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:50.190645 14464 raft_consensus.cc:2454] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 3.
I20240501 20:09:50.190994 14543 raft_consensus.cc:2454] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 3.
I20240501 20:09:50.191083 14583 leader_election.cc:302] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:50.191474 15193 raft_consensus.cc:2781] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:09:50.191648 15193 raft_consensus.cc:686] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 3 LEADER]: Becoming Leader. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:50.191933 15193 consensus_queue.cc:237] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:50.195466 14368 catalog_manager.cc:5484] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: term changed from 2 to 3, leader changed from 3da749f295434bceb714fb91c9598d45 (127.10.219.130) to d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131). New cstate: current_term: 3 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:50.240820 15128 raft_consensus.cc:979] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c: : Instructing follower 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:50.241039 15128 raft_consensus.cc:1067] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Signalling peer 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:50.241672 14543 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:35668
I20240501 20:09:50.241926 14543 raft_consensus.cc:482] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:50.242046 14543 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:50.244815 14543 raft_consensus.cc:504] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:50.245517 14543 leader_election.cc:288] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 3 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:50.245939 14464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:50.245991 14615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:50.246168 14464 raft_consensus.cc:3032] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:09:50.246227 14615 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:50.246268 14464 raft_consensus.cc:729] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:50.246613 14464 consensus_queue.cc:260] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:50.247015 14464 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:50.249382 14615 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 3.
I20240501 20:09:50.249560 14464 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 3.
I20240501 20:09:50.249780 14509 leader_election.cc:302] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:50.250105 15176 raft_consensus.cc:2781] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:09:50.250288 15176 raft_consensus.cc:686] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 3 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:50.250617 15176 consensus_queue.cc:237] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:50.253684 14368 catalog_manager.cc:5484] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 2 to 3, leader changed from 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 3 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:50.513770 14544 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer 12317e44612647c9bf2e942d55a89bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 20:09:50.514559 15197 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:50.515043 14464 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Refusing update from remote peer 12317e44612647c9bf2e942d55a89bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 20:09:50.515601 15198 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:50.572242 14544 raft_consensus.cc:1261] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240501 20:09:50.572963 15193 consensus_queue.cc:1035] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:50.593467 14464 raft_consensus.cc:1261] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240501 20:09:50.594766 15143 consensus_queue.cc:1035] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:50.688488 14615 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240501 20:09:50.689101 15176 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:50.690599 14464 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240501 20:09:50.691146 15176 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:50.710791 14543 raft_consensus.cc:1261] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 20:09:50.711416 15182 consensus_queue.cc:1035] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:50.715831 14615 raft_consensus.cc:1261] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 20:09:50.716765 15212 consensus_queue.cc:1035] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:50.839668 15167 tablet_copy_client.cc:523] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:50.843586 15167 tablet_bootstrap.cc:492] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0: Bootstrap starting.
I20240501 20:09:50.883364 15167 tablet_bootstrap.cc:492] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=113 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:50.883806 15167 tablet_bootstrap.cc:492] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0: Bootstrap complete.
I20240501 20:09:50.884107 15167 ts_tablet_manager.cc:1367] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0: Time spent bootstrapping tablet: real 0.041s user 0.035s sys 0.005s
I20240501 20:09:50.884802 15167 raft_consensus.cc:348] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:50.885047 15167 raft_consensus.cc:729] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 62ad649017804e6faf6684ff1478d3b0, State: Initialized, Role: LEARNER
I20240501 20:09:50.885303 15167 consensus_queue.cc:260] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:50.885949 15167 ts_tablet_manager.cc:1398] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240501 20:09:50.886765 14555 tablet_copy_service.cc:342] P 3da749f295434bceb714fb91c9598d45: Request end of tablet copy session 62ad649017804e6faf6684ff1478d3b0-9784e2a5d4324c63b42d41fe0fde222b received from {username='slave'} at 127.0.0.1:47562
I20240501 20:09:50.886898 14555 tablet_copy_service.cc:433] P 3da749f295434bceb714fb91c9598d45: ending tablet copy session 62ad649017804e6faf6684ff1478d3b0-9784e2a5d4324c63b42d41fe0fde222b on tablet 9784e2a5d4324c63b42d41fe0fde222b with peer 62ad649017804e6faf6684ff1478d3b0
I20240501 20:09:51.395570 14932 raft_consensus.cc:1203] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240501 20:09:51.779807 15219 raft_consensus.cc:1050] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: attempting to promote NON_VOTER 62ad649017804e6faf6684ff1478d3b0 to VOTER
I20240501 20:09:51.780647 15219 consensus_queue.cc:237] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } }
I20240501 20:09:51.783536 14615 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:51.784005 14467 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:51.784487 15220 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:51.784883 15210 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:51.785969 14932 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 1 LEARNER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 20:09:51.786525 15219 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:51.791352 14422 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:51.794896 14617 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b90dffdaa47425883abd651fe391006"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
mode: GRACEFUL
new_leader_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:34480
I20240501 20:09:51.795257 14617 raft_consensus.cc:595] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 3 LEADER]: Received request to transfer leadership to TS 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:51.796731 14467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
new_leader_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:55242
I20240501 20:09:51.797022 14467 raft_consensus.cc:595] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 3 LEADER]: Received request to transfer leadership to TS 3da749f295434bceb714fb91c9598d45
I20240501 20:09:51.798656 14544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f729e4d0e8dd4afbac82612b41d1eb4a"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
new_leader_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:35692
I20240501 20:09:51.799104 14544 raft_consensus.cc:595] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Received request to transfer leadership to TS 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:51.799820 14422 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 20:09:51.800091 14422 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:51.803545 15219 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:51.804224 14615 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:51.808322 14467 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:51.808348 14359 catalog_manager.cc:5484] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index 26 to 27, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:51.809266 14932 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:51.821516 14544 consensus_queue.cc:237] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } }
I20240501 20:09:51.823604 14932 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 20:09:51.823904 14467 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 20:09:51.824151 15223 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:51.825124 15223 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:51.826575 15219 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:51.827976 14467 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:51.830082 14355 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9784e2a5d4324c63b42d41fe0fde222b with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:51.830891 14932 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:51.831445 14361 catalog_manager.cc:5484] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index 27 to 28, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:51.835722 14607 tablet_service.cc:1514] Processing DeleteTablet for tablet 9784e2a5d4324c63b42d41fe0fde222b with delete_type TABLET_DATA_TOMBSTONED (TS d66b9c8dab924baa8463d26f5f25b397 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:38258
I20240501 20:09:51.836418 15231 tablet_replica.cc:330] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397: stopping tablet replica
I20240501 20:09:51.836900 15231 raft_consensus.cc:2229] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:51.837136 15231 raft_consensus.cc:2258] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:51.838008 15231 ts_tablet_manager.cc:1875] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:51.842821 15231 ts_tablet_manager.cc:1888] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240501 20:09:51.842962 15231 log.cc:1196] T 9784e2a5d4324c63b42d41fe0fde222b P d66b9c8dab924baa8463d26f5f25b397: Deleting WAL directory at /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-2-root/wals/9784e2a5d4324c63b42d41fe0fde222b
I20240501 20:09:51.843756 14353 catalog_manager.cc:4830] TS d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403): tablet 9784e2a5d4324c63b42d41fe0fde222b (table test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]) successfully deleted
I20240501 20:09:51.976563 15232 raft_consensus.cc:979] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c: : Instructing follower 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:51.976732 15232 raft_consensus.cc:1067] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 3 LEADER]: Signalling peer 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:51.977389 14543 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:35668
I20240501 20:09:51.977628 14543 raft_consensus.cc:482] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:51.977725 14543 raft_consensus.cc:3037] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:09:51.980072 14543 raft_consensus.cc:504] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:51.980844 14543 leader_election.cc:288] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 4 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:51.981386 14615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 4 candidate_status { last_received { term: 3 index: 26 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:51.981386 14467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8ec6baf414c598a80753fe882d9a0" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 4 candidate_status { last_received { term: 3 index: 26 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:51.981616 14615 raft_consensus.cc:3037] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:09:51.981616 14467 raft_consensus.cc:3032] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:09:51.981729 14467 raft_consensus.cc:729] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:51.981997 14467 consensus_queue.cc:260] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 3.26, Last appended by leader: 26, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:51.982409 14467 raft_consensus.cc:3037] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:09:51.984810 14615 raft_consensus.cc:2454] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 4.
I20240501 20:09:51.985046 14467 raft_consensus.cc:2454] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 4.
I20240501 20:09:51.985610 14509 leader_election.cc:302] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:51.985939 15219 raft_consensus.cc:2781] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:09:51.986084 15219 raft_consensus.cc:686] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 4 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:51.986367 15219 consensus_queue.cc:237] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 3.26, Last appended by leader: 26, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:51.989252 14361 catalog_manager.cc:5484] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 3 to 4, leader changed from 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 4 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:52.045761 15220 raft_consensus.cc:979] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45: : Instructing follower 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:52.045948 15220 raft_consensus.cc:1067] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Signalling peer 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:52.046660 14467 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f729e4d0e8dd4afbac82612b41d1eb4a"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:44456
I20240501 20:09:52.046865 14467 raft_consensus.cc:482] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:52.046957 14467 raft_consensus.cc:3037] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:52.049046 14467 raft_consensus.cc:504] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:52.049768 14467 leader_election.cc:288] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 2 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:52.050334 14615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f729e4d0e8dd4afbac82612b41d1eb4a" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:52.050328 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f729e4d0e8dd4afbac82612b41d1eb4a" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:52.050561 14615 raft_consensus.cc:3037] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:52.050561 14543 raft_consensus.cc:3032] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:52.050705 14543 raft_consensus.cc:729] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:52.051016 14543 consensus_queue.cc:260] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:52.051373 14543 raft_consensus.cc:3037] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:52.053587 14615 raft_consensus.cc:2454] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 2.
I20240501 20:09:52.054005 14543 raft_consensus.cc:2454] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 2.
I20240501 20:09:52.054121 14434 leader_election.cc:302] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:52.054435 15232 raft_consensus.cc:2781] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:52.054582 15232 raft_consensus.cc:686] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Becoming Leader. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:52.054849 15232 consensus_queue.cc:237] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:52.057600 14361 catalog_manager.cc:5484] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: term changed from 1 to 2, leader changed from 3da749f295434bceb714fb91c9598d45 (127.10.219.130) to 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129). New cstate: current_term: 2 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:52.102941 15215 raft_consensus.cc:979] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397: : Instructing follower 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:52.103092 15215 raft_consensus.cc:1067] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 3 LEADER]: Signalling peer 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:52.103849 14467 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b90dffdaa47425883abd651fe391006"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:44462
I20240501 20:09:52.104090 14467 raft_consensus.cc:482] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:52.104180 14467 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:09:52.106603 14467 raft_consensus.cc:504] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:52.107336 14467 leader_election.cc:288] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 4 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:52.107889 14615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 4 candidate_status { last_received { term: 3 index: 26 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:52.107887 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 4 candidate_status { last_received { term: 3 index: 26 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:52.108088 14615 raft_consensus.cc:3032] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:09:52.108096 14543 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:09:52.108165 14615 raft_consensus.cc:729] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:52.108433 14615 consensus_queue.cc:260] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 3.26, Last appended by leader: 26, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:52.108744 14615 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:09:52.110420 14543 raft_consensus.cc:2454] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 4.
I20240501 20:09:52.110759 14615 raft_consensus.cc:2454] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 4.
I20240501 20:09:52.110939 14436 leader_election.cc:302] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:52.111296 15232 raft_consensus.cc:2781] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:09:52.111439 15232 raft_consensus.cc:686] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 4 LEADER]: Becoming Leader. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:52.111711 15232 consensus_queue.cc:237] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 3.26, Last appended by leader: 26, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:52.114748 14361 catalog_manager.cc:5484] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: term changed from 3 to 4, leader changed from d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) to 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129). New cstate: current_term: 4 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:52.405352 14615 raft_consensus.cc:1261] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 4 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 4 index: 27. (index mismatch)
I20240501 20:09:52.406026 15220 consensus_queue.cc:1035] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:52.410418 14467 raft_consensus.cc:1261] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 4 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 4 index: 27. (index mismatch)
I20240501 20:09:52.410959 15220 consensus_queue.cc:1035] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:52.484462 14615 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 20:09:52.485222 14543 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 20:09:52.485127 15232 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 20:09:52.485747 15232 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 20:09:52.676982 14615 raft_consensus.cc:1261] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 4 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 4 index: 27. (index mismatch)
I20240501 20:09:52.677495 15218 consensus_queue.cc:1035] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:52.681329 14543 raft_consensus.cc:1261] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 4 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 4 index: 27. (index mismatch)
I20240501 20:09:52.681978 15244 consensus_queue.cc:1035] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:52.839852 14543 consensus_queue.cc:237] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 3.26, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:52.842754 14467 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 20:09:52.845499 15219 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20240501 20:09:52.846367 14615 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 20:09:52.847048 15219 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:52.851943 15219 raft_consensus.cc:2932] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 3 LEADER]: Committing config change with OpId 3.27: config changed from index -1 to 27, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
W20240501 20:09:52.853523 14509 consensus_peers.cc:485] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 -> Peer 12317e44612647c9bf2e942d55a89bfa (127.10.219.132:44999): Couldn't send request to peer 12317e44612647c9bf2e942d55a89bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd5f05007317454bb4e7bdefd81ccd90. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:52.854420 14467 raft_consensus.cc:2932] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Committing config change with OpId 3.27: config changed from index -1 to 27, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:52.854775 14615 raft_consensus.cc:2932] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Committing config change with OpId 3.27: config changed from index -1 to 27, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:52.858981 14543 consensus_queue.cc:237] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:52.860872 14361 catalog_manager.cc:5484] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: config changed from index -1 to 27, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New cstate: current_term: 3 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:52.863612 14932 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 20:09:52.863803 14467 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 20:09:52.864356 15220 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240501 20:09:52.864822 14509 consensus_peers.cc:485] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 -> Peer 12317e44612647c9bf2e942d55a89bfa (127.10.219.132:44999): Couldn't send request to peer 12317e44612647c9bf2e942d55a89bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9784e2a5d4324c63b42d41fe0fde222b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:52.865101 15247 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:52.882082 15247 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:52.882987 14467 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:52.892644 14364 catalog_manager.cc:5484] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: config changed from index 28 to 29, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New cstate: current_term: 1 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:52.892561 14932 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:52.920503 15006 consensus_queue.cc:237] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:52.927425 14615 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Refusing update from remote peer d1eb1382cb8f47b7aef9d61088b20974: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 20:09:52.928563 15181 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 20:09:52.933326 14543 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer d1eb1382cb8f47b7aef9d61088b20974: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 20:09:52.934198 15262 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.002s
W20240501 20:09:52.938009 14969 consensus_peers.cc:485] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 -> Peer 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133:46093): Couldn't send request to peer 62ad649017804e6faf6684ff1478d3b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45bd961f0ac04a4a812a818a864c902f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:52.939471 15181 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:52.940384 14543 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:52.943106 14615 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:52.945785 14364 catalog_manager.cc:5484] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index 28 to 30, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New cstate: current_term: 2 leader_uuid: "d1eb1382cb8f47b7aef9d61088b20974" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:52.956419 14857 consensus_queue.cc:237] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:52.966653 14467 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Refusing update from remote peer 12317e44612647c9bf2e942d55a89bfa: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 20:09:52.968626 15198 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240501 20:09:52.976673 14821 consensus_peers.cc:485] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa -> Peer d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134:46221): Couldn't send request to peer d1eb1382cb8f47b7aef9d61088b20974. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85e1a32af97d4a238310403e907092e0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:52.977342 14543 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer 12317e44612647c9bf2e942d55a89bfa: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 20:09:52.978085 15269 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 20:09:52.977694 15198 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:52.979249 14467 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:52.986594 14464 consensus_queue.cc:237] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:52.988237 14361 catalog_manager.cc:5484] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: config changed from index 28 to 30, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New cstate: current_term: 2 leader_uuid: "12317e44612647c9bf2e942d55a89bfa" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
W20240501 20:09:52.998128 14434 consensus_peers.cc:485] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c -> Peer d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134:46221): Couldn't send request to peer d1eb1382cb8f47b7aef9d61088b20974. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f729e4d0e8dd4afbac82612b41d1eb4a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:52.999388 14615 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 20:09:53.000365 15232 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:53.002645 14543 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 20:09:53.003450 15232 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:53.005472 15232 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:53.011801 14543 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:53.015751 14361 catalog_manager.cc:5484] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: config changed from index -1 to 26, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New cstate: current_term: 2 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:53.011801 14615 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:53.019178 14617 consensus_queue.cc:237] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:53.049750 14543 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240501 20:09:53.050809 15215 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 20:09:53.058917 14464 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240501 20:09:53.061136 15279 raft_consensus.cc:2932] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:53.067817 15278 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 20:09:53.068002 14543 raft_consensus.cc:2932] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:53.075984 14361 catalog_manager.cc:5484] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index -1 to 25, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New cstate: current_term: 1 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:53.082473 14464 raft_consensus.cc:2932] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
W20240501 20:09:53.090701 14821 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.219.129:33765, user_credentials={real_user=slave}} blocked reactor thread for 104317us
W20240501 20:09:53.096724 14823 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.219.130:40971, user_credentials={real_user=slave}} blocked reactor thread for 118913us
I20240501 20:09:53.100361 14543 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
W20240501 20:09:53.118286 14583 consensus_peers.cc:485] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 -> Peer 12317e44612647c9bf2e942d55a89bfa (127.10.219.132:44999): Couldn't send request to peer 12317e44612647c9bf2e942d55a89bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b3921d54966469c8de55002d0f93a10. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:53.305900 15282 ts_tablet_manager.cc:902] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa: Initiating tablet copy from peer 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:53.309074 15282 tablet_copy_client.cc:308] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.130:40971
I20240501 20:09:53.310024 14555 tablet_copy_service.cc:140] P 3da749f295434bceb714fb91c9598d45: Received BeginTabletCopySession request for tablet 9784e2a5d4324c63b42d41fe0fde222b from peer 12317e44612647c9bf2e942d55a89bfa ({username='slave'} at 127.0.0.1:47572)
I20240501 20:09:53.310191 14555 tablet_copy_service.cc:161] P 3da749f295434bceb714fb91c9598d45: Beginning new tablet copy session on tablet 9784e2a5d4324c63b42d41fe0fde222b from peer 12317e44612647c9bf2e942d55a89bfa at {username='slave'} at 127.0.0.1:47572: session id = 12317e44612647c9bf2e942d55a89bfa-9784e2a5d4324c63b42d41fe0fde222b
I20240501 20:09:53.313043 14555 tablet_copy_source_session.cc:205] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:53.314658 15282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9784e2a5d4324c63b42d41fe0fde222b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.320551 15282 tablet_copy_client.cc:783] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:53.320853 15282 tablet_copy_client.cc:647] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:53.322611 15286 tablet_copy_client.cc:1002] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:53.344853 15287 ts_tablet_manager.cc:902] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0: Initiating tablet copy from peer d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134:46221)
I20240501 20:09:53.349511 15287 tablet_copy_client.cc:308] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.134:46221
I20240501 20:09:53.358539 15016 tablet_copy_service.cc:140] P d1eb1382cb8f47b7aef9d61088b20974: Received BeginTabletCopySession request for tablet 45bd961f0ac04a4a812a818a864c902f from peer 62ad649017804e6faf6684ff1478d3b0 ({username='slave'} at 127.0.0.1:44648)
I20240501 20:09:53.358891 15016 tablet_copy_service.cc:161] P d1eb1382cb8f47b7aef9d61088b20974: Beginning new tablet copy session on tablet 45bd961f0ac04a4a812a818a864c902f from peer 62ad649017804e6faf6684ff1478d3b0 at {username='slave'} at 127.0.0.1:44648: session id = 62ad649017804e6faf6684ff1478d3b0-45bd961f0ac04a4a812a818a864c902f
I20240501 20:09:53.364190 15016 tablet_copy_source_session.cc:205] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:53.366535 15287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bd961f0ac04a4a812a818a864c902f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.376663 15287 tablet_copy_client.cc:783] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:53.377122 15287 tablet_copy_client.cc:647] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:53.379280 15291 tablet_copy_client.cc:1002] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:53.383071 15293 ts_tablet_manager.cc:902] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa: Initiating tablet copy from peer 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:53.390933 15293 tablet_copy_client.cc:308] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.130:40971
I20240501 20:09:53.391564 14555 tablet_copy_service.cc:140] P 3da749f295434bceb714fb91c9598d45: Received BeginTabletCopySession request for tablet cd5f05007317454bb4e7bdefd81ccd90 from peer 12317e44612647c9bf2e942d55a89bfa ({username='slave'} at 127.0.0.1:47572)
I20240501 20:09:53.391731 14555 tablet_copy_service.cc:161] P 3da749f295434bceb714fb91c9598d45: Beginning new tablet copy session on tablet cd5f05007317454bb4e7bdefd81ccd90 from peer 12317e44612647c9bf2e942d55a89bfa at {username='slave'} at 127.0.0.1:47572: session id = 12317e44612647c9bf2e942d55a89bfa-cd5f05007317454bb4e7bdefd81ccd90
I20240501 20:09:53.394640 14555 tablet_copy_source_session.cc:205] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:53.396392 15293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd5f05007317454bb4e7bdefd81ccd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.406493 15293 tablet_copy_client.cc:783] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:53.406895 15293 tablet_copy_client.cc:647] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:53.408731 15297 tablet_copy_client.cc:1002] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:53.457223 15298 ts_tablet_manager.cc:902] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974: Initiating tablet copy from peer 12317e44612647c9bf2e942d55a89bfa (127.10.219.132:44999)
I20240501 20:09:53.467851 15298 tablet_copy_client.cc:308] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.132:44999
I20240501 20:09:53.485656 14867 tablet_copy_service.cc:140] P 12317e44612647c9bf2e942d55a89bfa: Received BeginTabletCopySession request for tablet 85e1a32af97d4a238310403e907092e0 from peer d1eb1382cb8f47b7aef9d61088b20974 ({username='slave'} at 127.0.0.1:36566)
I20240501 20:09:53.485901 14867 tablet_copy_service.cc:161] P 12317e44612647c9bf2e942d55a89bfa: Beginning new tablet copy session on tablet 85e1a32af97d4a238310403e907092e0 from peer d1eb1382cb8f47b7aef9d61088b20974 at {username='slave'} at 127.0.0.1:36566: session id = d1eb1382cb8f47b7aef9d61088b20974-85e1a32af97d4a238310403e907092e0
I20240501 20:09:53.488606 14867 tablet_copy_source_session.cc:205] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:53.489918 15298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e1a32af97d4a238310403e907092e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.496029 15298 tablet_copy_client.cc:783] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:53.496313 15298 tablet_copy_client.cc:647] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:53.497947 15302 tablet_copy_client.cc:1002] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:53.603413 15305 ts_tablet_manager.cc:902] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974: Initiating tablet copy from peer 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765)
I20240501 20:09:53.604915 15305 tablet_copy_client.cc:308] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.129:33765
I20240501 20:09:53.605535 14480 tablet_copy_service.cc:140] P 8d368a895e2a48c8a17b384c0739e47c: Received BeginTabletCopySession request for tablet f729e4d0e8dd4afbac82612b41d1eb4a from peer d1eb1382cb8f47b7aef9d61088b20974 ({username='slave'} at 127.0.0.1:44536)
I20240501 20:09:53.605662 14480 tablet_copy_service.cc:161] P 8d368a895e2a48c8a17b384c0739e47c: Beginning new tablet copy session on tablet f729e4d0e8dd4afbac82612b41d1eb4a from peer d1eb1382cb8f47b7aef9d61088b20974 at {username='slave'} at 127.0.0.1:44536: session id = d1eb1382cb8f47b7aef9d61088b20974-f729e4d0e8dd4afbac82612b41d1eb4a
I20240501 20:09:53.608454 14480 tablet_copy_source_session.cc:205] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:53.609774 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f729e4d0e8dd4afbac82612b41d1eb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.617050 15305 tablet_copy_client.cc:783] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:53.617375 15305 tablet_copy_client.cc:647] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:53.618974 15309 tablet_copy_client.cc:1002] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:53.622004 15310 ts_tablet_manager.cc:902] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa: Initiating tablet copy from peer d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:53.630103 15310 tablet_copy_client.cc:308] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.131:39403
I20240501 20:09:53.653066 14629 tablet_copy_service.cc:140] P d66b9c8dab924baa8463d26f5f25b397: Received BeginTabletCopySession request for tablet 4b3921d54966469c8de55002d0f93a10 from peer 12317e44612647c9bf2e942d55a89bfa ({username='slave'} at 127.0.0.1:34520)
I20240501 20:09:53.653405 14629 tablet_copy_service.cc:161] P d66b9c8dab924baa8463d26f5f25b397: Beginning new tablet copy session on tablet 4b3921d54966469c8de55002d0f93a10 from peer 12317e44612647c9bf2e942d55a89bfa at {username='slave'} at 127.0.0.1:34520: session id = 12317e44612647c9bf2e942d55a89bfa-4b3921d54966469c8de55002d0f93a10
I20240501 20:09:53.656370 14629 tablet_copy_source_session.cc:205] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:53.657774 15310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b3921d54966469c8de55002d0f93a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.665273 15310 tablet_copy_client.cc:783] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:53.665947 15310 tablet_copy_client.cc:647] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:53.668259 15314 tablet_copy_client.cc:1002] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:53.800506 14422 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:53.804013 14464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b90dffdaa47425883abd651fe391006"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
new_leader_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:55242
I20240501 20:09:53.804389 14464 raft_consensus.cc:595] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 4 LEADER]: Received request to transfer leadership to TS 3da749f295434bceb714fb91c9598d45
I20240501 20:09:53.805789 14464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "baf26068e4c246baa73afccb4e93c847"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
new_leader_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:55242
I20240501 20:09:53.806068 14464 raft_consensus.cc:595] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Received request to transfer leadership to TS d66b9c8dab924baa8463d26f5f25b397
I20240501 20:09:53.807246 14543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
new_leader_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:35692
I20240501 20:09:53.807440 14543 raft_consensus.cc:595] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 3 LEADER]: Received request to transfer leadership to TS d66b9c8dab924baa8463d26f5f25b397
I20240501 20:09:53.807924 14422 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 20:09:53.808176 14422 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:53.846699 15232 raft_consensus.cc:979] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c: : Instructing follower d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:53.846861 15232 raft_consensus.cc:1067] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Signalling peer d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:53.847481 14615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "baf26068e4c246baa73afccb4e93c847"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:38264
I20240501 20:09:53.847688 14615 raft_consensus.cc:482] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:53.847788 14615 raft_consensus.cc:3037] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:53.850519 14615 raft_consensus.cc:504] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:53.851248 14615 leader_election.cc:288] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 2 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134:46221)
I20240501 20:09:53.851769 14464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baf26068e4c246baa73afccb4e93c847" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:53.851962 14464 raft_consensus.cc:3032] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:53.852036 14464 raft_consensus.cc:729] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:53.852300 14464 consensus_queue.cc:260] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:53.852584 14464 raft_consensus.cc:3037] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:53.854512 14464 raft_consensus.cc:2454] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 2.
I20240501 20:09:53.855351 14583 leader_election.cc:302] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:53.855661 15279 raft_consensus.cc:2781] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:53.856132 15279 raft_consensus.cc:686] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Becoming Leader. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:53.856448 15279 consensus_queue.cc:237] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:53.859573 14361 catalog_manager.cc:5484] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: term changed from 1 to 2, leader changed from 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) to d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131). New cstate: current_term: 2 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:53.862747 15006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baf26068e4c246baa73afccb4e93c847" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "d1eb1382cb8f47b7aef9d61088b20974"
I20240501 20:09:53.862977 15006 raft_consensus.cc:3037] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:53.866273 15006 raft_consensus.cc:2454] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 2.
I20240501 20:09:54.226748 15247 raft_consensus.cc:979] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45: : Instructing follower d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:54.226976 15318 raft_consensus.cc:1067] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 3 LEADER]: Signalling peer d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:54.227643 14617 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:38276
I20240501 20:09:54.227852 14617 raft_consensus.cc:482] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:54.227964 14617 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:09:54.230923 14617 raft_consensus.cc:504] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:54.231986 14617 leader_election.cc:288] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 4 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:54.232153 14465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:54.232357 14465 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:09:54.232668 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:54.232856 14543 raft_consensus.cc:3032] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:09:54.232918 14543 raft_consensus.cc:729] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:54.233191 14543 consensus_queue.cc:260] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:54.233640 14543 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:09:54.235718 14465 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 4.
I20240501 20:09:54.236268 14583 leader_election.cc:302] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:54.236636 14543 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 4.
I20240501 20:09:54.237908 15329 raft_consensus.cc:2781] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:09:54.238082 15329 raft_consensus.cc:686] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 4 LEADER]: Becoming Leader. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:54.238435 15329 consensus_queue.cc:237] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:54.242048 14368 catalog_manager.cc:5484] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: term changed from 3 to 4, leader changed from 3da749f295434bceb714fb91c9598d45 (127.10.219.130) to d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131). New cstate: current_term: 4 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:54.272411 15321 raft_consensus.cc:979] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c: : Instructing follower 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:54.272610 15321 raft_consensus.cc:1067] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 4 LEADER]: Signalling peer 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:54.273353 14543 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b90dffdaa47425883abd651fe391006"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:35668
I20240501 20:09:54.273600 14543 raft_consensus.cc:482] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:54.273727 14543 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:09:54.277177 14543 raft_consensus.cc:504] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:54.277889 14543 leader_election.cc:288] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 5 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:54.278334 14615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 5 candidate_status { last_received { term: 4 index: 27 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:54.278342 14465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90dffdaa47425883abd651fe391006" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 5 candidate_status { last_received { term: 4 index: 27 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:54.278611 14615 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:09:54.278623 14465 raft_consensus.cc:3032] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:09:54.278720 14465 raft_consensus.cc:729] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:54.278997 14465 consensus_queue.cc:260] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 4.27, Last appended by leader: 27, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:54.279356 14465 raft_consensus.cc:3037] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:09:54.281036 14615 raft_consensus.cc:2454] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 5.
I20240501 20:09:54.281670 14465 raft_consensus.cc:2454] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 5.
I20240501 20:09:54.281642 14509 leader_election.cc:302] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:54.282117 15317 raft_consensus.cc:2781] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:09:54.282279 15317 raft_consensus.cc:686] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 5 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:54.282640 15317 consensus_queue.cc:237] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 4.27, Last appended by leader: 27, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } }
I20240501 20:09:54.286401 14368 catalog_manager.cc:5484] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 4 to 5, leader changed from 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 5 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:54.308862 15006 raft_consensus.cc:1261] T baf26068e4c246baa73afccb4e93c847 P d1eb1382cb8f47b7aef9d61088b20974 [term 2 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 20:09:54.309535 15329 consensus_queue.cc:1035] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:54.314427 14465 raft_consensus.cc:1261] T baf26068e4c246baa73afccb4e93c847 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 20:09:54.315055 15323 consensus_queue.cc:1035] T baf26068e4c246baa73afccb4e93c847 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:54.675210 14543 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 4 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240501 20:09:54.675837 15323 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240501 20:09:54.680923 14583 consensus_peers.cc:485] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 -> Peer 12317e44612647c9bf2e942d55a89bfa (127.10.219.132:44999): Couldn't send request to peer 12317e44612647c9bf2e942d55a89bfa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:54.681281 14465 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 4 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240501 20:09:54.683480 15323 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:54.837409 14464 raft_consensus.cc:1261] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 5 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 4 index: 27. Preceding OpId from leader: term: 5 index: 28. (index mismatch)
I20240501 20:09:54.837982 15317 consensus_queue.cc:1035] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:54.843010 14615 raft_consensus.cc:1261] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 5 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 4 index: 27. Preceding OpId from leader: term: 5 index: 28. (index mismatch)
I20240501 20:09:54.843586 15317 consensus_queue.cc:1035] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:55.324072 15282 tablet_copy_client.cc:523] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:55.327880 15282 tablet_bootstrap.cc:492] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa: Bootstrap starting.
I20240501 20:09:55.368639 15282 tablet_bootstrap.cc:492] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=113 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:55.369096 15282 tablet_bootstrap.cc:492] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa: Bootstrap complete.
I20240501 20:09:55.369484 15282 ts_tablet_manager.cc:1367] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa: Time spent bootstrapping tablet: real 0.042s user 0.038s sys 0.005s
I20240501 20:09:55.370285 15282 raft_consensus.cc:348] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:55.370597 15282 raft_consensus.cc:729] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12317e44612647c9bf2e942d55a89bfa, State: Initialized, Role: LEARNER
I20240501 20:09:55.370797 15282 consensus_queue.cc:260] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:55.371316 15282 ts_tablet_manager.cc:1398] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:55.371982 14555 tablet_copy_service.cc:342] P 3da749f295434bceb714fb91c9598d45: Request end of tablet copy session 12317e44612647c9bf2e942d55a89bfa-9784e2a5d4324c63b42d41fe0fde222b received from {username='slave'} at 127.0.0.1:47572
I20240501 20:09:55.372104 14555 tablet_copy_service.cc:433] P 3da749f295434bceb714fb91c9598d45: ending tablet copy session 12317e44612647c9bf2e942d55a89bfa-9784e2a5d4324c63b42d41fe0fde222b on tablet 9784e2a5d4324c63b42d41fe0fde222b with peer 12317e44612647c9bf2e942d55a89bfa
I20240501 20:09:55.382372 15287 tablet_copy_client.cc:523] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:55.385672 15287 tablet_bootstrap.cc:492] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0: Bootstrap starting.
I20240501 20:09:55.409961 15293 tablet_copy_client.cc:523] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:55.412907 15293 tablet_bootstrap.cc:492] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa: Bootstrap starting.
I20240501 20:09:55.423877 15287 tablet_bootstrap.cc:492] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:55.424345 15287 tablet_bootstrap.cc:492] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0: Bootstrap complete.
I20240501 20:09:55.424651 15287 ts_tablet_manager.cc:1367] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0: Time spent bootstrapping tablet: real 0.039s user 0.038s sys 0.001s
I20240501 20:09:55.425465 15287 raft_consensus.cc:348] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:55.425731 15287 raft_consensus.cc:729] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 62ad649017804e6faf6684ff1478d3b0, State: Initialized, Role: LEARNER
I20240501 20:09:55.426003 15287 consensus_queue.cc:260] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:55.426601 15287 ts_tablet_manager.cc:1398] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:09:55.427351 15016 tablet_copy_service.cc:342] P d1eb1382cb8f47b7aef9d61088b20974: Request end of tablet copy session 62ad649017804e6faf6684ff1478d3b0-45bd961f0ac04a4a812a818a864c902f received from {username='slave'} at 127.0.0.1:44648
I20240501 20:09:55.427466 15016 tablet_copy_service.cc:433] P d1eb1382cb8f47b7aef9d61088b20974: ending tablet copy session 62ad649017804e6faf6684ff1478d3b0-45bd961f0ac04a4a812a818a864c902f on tablet 45bd961f0ac04a4a812a818a864c902f with peer 62ad649017804e6faf6684ff1478d3b0
I20240501 20:09:55.447985 15293 tablet_bootstrap.cc:492] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=95 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:55.448426 15293 tablet_bootstrap.cc:492] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa: Bootstrap complete.
I20240501 20:09:55.448688 15293 ts_tablet_manager.cc:1367] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa: Time spent bootstrapping tablet: real 0.036s user 0.032s sys 0.002s
I20240501 20:09:55.449316 15293 raft_consensus.cc:348] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:55.449520 15293 raft_consensus.cc:729] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 12317e44612647c9bf2e942d55a89bfa, State: Initialized, Role: LEARNER
I20240501 20:09:55.449745 15293 consensus_queue.cc:260] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:55.450244 15293 ts_tablet_manager.cc:1398] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:55.451112 14555 tablet_copy_service.cc:342] P 3da749f295434bceb714fb91c9598d45: Request end of tablet copy session 12317e44612647c9bf2e942d55a89bfa-cd5f05007317454bb4e7bdefd81ccd90 received from {username='slave'} at 127.0.0.1:47572
I20240501 20:09:55.451246 14555 tablet_copy_service.cc:433] P 3da749f295434bceb714fb91c9598d45: ending tablet copy session 12317e44612647c9bf2e942d55a89bfa-cd5f05007317454bb4e7bdefd81ccd90 on tablet cd5f05007317454bb4e7bdefd81ccd90 with peer 12317e44612647c9bf2e942d55a89bfa
I20240501 20:09:55.499097 15298 tablet_copy_client.cc:523] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:55.502985 15298 tablet_bootstrap.cc:492] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974: Bootstrap starting.
I20240501 20:09:55.543869 15298 tablet_bootstrap.cc:492] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:55.544302 15298 tablet_bootstrap.cc:492] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974: Bootstrap complete.
I20240501 20:09:55.544582 15298 ts_tablet_manager.cc:1367] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974: Time spent bootstrapping tablet: real 0.042s user 0.038s sys 0.002s
I20240501 20:09:55.545243 15298 raft_consensus.cc:348] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:55.545507 15298 raft_consensus.cc:729] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d1eb1382cb8f47b7aef9d61088b20974, State: Initialized, Role: LEARNER
I20240501 20:09:55.545745 15298 consensus_queue.cc:260] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:55.546447 15298 ts_tablet_manager.cc:1398] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:55.547269 14867 tablet_copy_service.cc:342] P 12317e44612647c9bf2e942d55a89bfa: Request end of tablet copy session d1eb1382cb8f47b7aef9d61088b20974-85e1a32af97d4a238310403e907092e0 received from {username='slave'} at 127.0.0.1:36566
I20240501 20:09:55.547375 14867 tablet_copy_service.cc:433] P 12317e44612647c9bf2e942d55a89bfa: ending tablet copy session d1eb1382cb8f47b7aef9d61088b20974-85e1a32af97d4a238310403e907092e0 on tablet 85e1a32af97d4a238310403e907092e0 with peer d1eb1382cb8f47b7aef9d61088b20974
I20240501 20:09:55.619657 15305 tablet_copy_client.cc:523] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:55.623150 15305 tablet_bootstrap.cc:492] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974: Bootstrap starting.
I20240501 20:09:55.646451 14857 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 3 LEARNER]: Advancing to term 4
I20240501 20:09:55.665936 15305 tablet_bootstrap.cc:492] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:55.666486 15305 tablet_bootstrap.cc:492] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974: Bootstrap complete.
I20240501 20:09:55.666756 15305 ts_tablet_manager.cc:1367] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974: Time spent bootstrapping tablet: real 0.044s user 0.040s sys 0.004s
I20240501 20:09:55.667439 15305 raft_consensus.cc:348] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:55.667673 15305 raft_consensus.cc:729] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d1eb1382cb8f47b7aef9d61088b20974, State: Initialized, Role: LEARNER
I20240501 20:09:55.667871 15305 consensus_queue.cc:260] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:55.668402 15305 ts_tablet_manager.cc:1398] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:55.668929 15310 tablet_copy_client.cc:523] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:55.669472 14480 tablet_copy_service.cc:342] P 8d368a895e2a48c8a17b384c0739e47c: Request end of tablet copy session d1eb1382cb8f47b7aef9d61088b20974-f729e4d0e8dd4afbac82612b41d1eb4a received from {username='slave'} at 127.0.0.1:44536
I20240501 20:09:55.669591 14480 tablet_copy_service.cc:433] P 8d368a895e2a48c8a17b384c0739e47c: ending tablet copy session d1eb1382cb8f47b7aef9d61088b20974-f729e4d0e8dd4afbac82612b41d1eb4a on tablet f729e4d0e8dd4afbac82612b41d1eb4a with peer d1eb1382cb8f47b7aef9d61088b20974
I20240501 20:09:55.673100 15310 tablet_bootstrap.cc:492] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa: Bootstrap starting.
I20240501 20:09:55.711781 15310 tablet_bootstrap.cc:492] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=85 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:55.712201 15310 tablet_bootstrap.cc:492] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa: Bootstrap complete.
I20240501 20:09:55.712512 15310 ts_tablet_manager.cc:1367] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa: Time spent bootstrapping tablet: real 0.040s user 0.035s sys 0.004s
I20240501 20:09:55.713243 15310 raft_consensus.cc:348] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:55.713469 15310 raft_consensus.cc:729] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12317e44612647c9bf2e942d55a89bfa, State: Initialized, Role: LEARNER
I20240501 20:09:55.713660 15310 consensus_queue.cc:260] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:55.714124 15310 ts_tablet_manager.cc:1398] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:55.714792 14629 tablet_copy_service.cc:342] P d66b9c8dab924baa8463d26f5f25b397: Request end of tablet copy session 12317e44612647c9bf2e942d55a89bfa-4b3921d54966469c8de55002d0f93a10 received from {username='slave'} at 127.0.0.1:34520
I20240501 20:09:55.714905 14629 tablet_copy_service.cc:433] P d66b9c8dab924baa8463d26f5f25b397: ending tablet copy session 12317e44612647c9bf2e942d55a89bfa-4b3921d54966469c8de55002d0f93a10 on tablet 4b3921d54966469c8de55002d0f93a10 with peer 12317e44612647c9bf2e942d55a89bfa
W20240501 20:09:55.753326 14491 debug-util.cc:398] Leaking SignalData structure 0x603002b2a8a0 after lost signal to thread 15318
I20240501 20:09:55.754299 14857 raft_consensus.cc:1203] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240501 20:09:55.808565 14422 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:55.812330 14617 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
mode: GRACEFUL
new_leader_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:34480
I20240501 20:09:55.813555 14617 raft_consensus.cc:595] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 1 LEADER]: Received request to transfer leadership to TS 3da749f295434bceb714fb91c9598d45
I20240501 20:09:55.814936 14543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
new_leader_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:35692
I20240501 20:09:55.815182 14543 raft_consensus.cc:595] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Received request to transfer leadership to TS 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:55.816603 14617 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
mode: GRACEFUL
new_leader_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:34480
I20240501 20:09:55.816992 14617 raft_consensus.cc:595] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 4 LEADER]: Received request to transfer leadership to TS 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:55.817591 14422 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 20:09:55.818184 14422 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:55.829994 14932 raft_consensus.cc:1203] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240501 20:09:55.830329 14857 raft_consensus.cc:1203] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240501 20:09:55.870215 15005 raft_consensus.cc:1203] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240501 20:09:55.939122 15356 raft_consensus.cc:979] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397: : Instructing follower 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:55.939368 15356 raft_consensus.cc:1067] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 1 LEADER]: Signalling peer 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:55.940080 14543 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:35670
I20240501 20:09:55.940315 14543 raft_consensus.cc:482] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:55.940434 14543 raft_consensus.cc:3037] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:55.942785 14543 raft_consensus.cc:504] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:55.946137 14543 leader_election.cc:288] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 2 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:55.946238 14615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:55.946532 14615 raft_consensus.cc:3032] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:55.946621 14615 raft_consensus.cc:729] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:55.947091 14464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:55.946988 14615 consensus_queue.cc:260] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:55.947263 14464 raft_consensus.cc:3037] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:55.947602 14615 raft_consensus.cc:3037] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:55.949430 14464 raft_consensus.cc:2454] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 2.
I20240501 20:09:55.949822 14509 leader_election.cc:302] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:55.950083 15317 raft_consensus.cc:2781] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:55.951592 14615 raft_consensus.cc:2454] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 2.
I20240501 20:09:55.959507 15279 raft_consensus.cc:979] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397: : Instructing follower 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:55.959780 15279 raft_consensus.cc:1067] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 4 LEADER]: Signalling peer 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:55.960757 14464 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:44462
I20240501 20:09:55.961012 14464 raft_consensus.cc:482] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:55.961122 14464 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:09:55.962831 15317 raft_consensus.cc:686] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 2 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:55.963146 15317 consensus_queue.cc:237] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:55.964861 14464 raft_consensus.cc:504] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:55.966104 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:55.966336 14543 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:09:55.967325 14368 catalog_manager.cc:5484] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 1 to 2, leader changed from d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 2 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:55.968425 14543 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 5.
I20240501 20:09:55.968920 14436 leader_election.cc:302] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:55.969221 15357 raft_consensus.cc:2781] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:09:55.969359 15357 raft_consensus.cc:686] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 5 LEADER]: Becoming Leader. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:55.969604 15357 consensus_queue.cc:237] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 28, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:55.971249 14464 leader_election.cc:288] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 5 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:55.971629 14615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:55.972100 14615 raft_consensus.cc:3032] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:09:55.972230 14615 raft_consensus.cc:729] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:55.972693 14615 consensus_queue.cc:260] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 28, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:55.973407 14615 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:09:55.978070 14615 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 5.
I20240501 20:09:55.980957 14361 catalog_manager.cc:5484] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: term changed from 4 to 5, leader changed from d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) to 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129). New cstate: current_term: 5 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:55.993899 14543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35656
I20240501 20:09:55.994112 14543 raft_consensus.cc:595] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 2 LEADER]: Received request to transfer leadership
I20240501 20:09:55.996202 14464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44444
I20240501 20:09:55.996398 14464 raft_consensus.cc:595] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 5 LEADER]: Received request to transfer leadership
I20240501 20:09:56.003921 14543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35656
I20240501 20:09:56.004135 14543 raft_consensus.cc:595] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 2 LEADER]: Received request to transfer leadership
W20240501 20:09:56.004580 14355 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 4b3921d54966469c8de55002d0f93a10 already in progress [suppressed 1 similar messages]
I20240501 20:09:56.005786 14543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35656
I20240501 20:09:56.005972 14543 raft_consensus.cc:595] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 2 LEADER]: Received request to transfer leadership
W20240501 20:09:56.006649 14416 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 4b3921d54966469c8de55002d0f93a10 already in progress
W20240501 20:09:56.006759 14416 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 4b3921d54966469c8de55002d0f93a10 already in progress
I20240501 20:09:56.008867 14464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44444
I20240501 20:09:56.009112 14464 raft_consensus.cc:595] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 5 LEADER]: Received request to transfer leadership
I20240501 20:09:56.010221 14464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44444
I20240501 20:09:56.010501 14464 raft_consensus.cc:595] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 5 LEADER]: Received request to transfer leadership
W20240501 20:09:56.011065 14416 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for cd5f05007317454bb4e7bdefd81ccd90 already in progress
W20240501 20:09:56.011145 14416 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for cd5f05007317454bb4e7bdefd81ccd90 already in progress
I20240501 20:09:56.072922 15005 raft_consensus.cc:1203] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.25->[2.26-2.26] Dedup: 2.26->[]
I20240501 20:09:56.088052 15358 raft_consensus.cc:979] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: : Instructing follower 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:56.088253 15358 raft_consensus.cc:1067] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Signalling peer 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:56.088956 14464 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:44456
I20240501 20:09:56.089205 14464 raft_consensus.cc:482] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:56.089322 14464 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:56.093071 14464 raft_consensus.cc:504] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:56.094405 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:56.094658 14543 raft_consensus.cc:3032] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:56.094739 14543 raft_consensus.cc:729] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:56.094851 14932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "62ad649017804e6faf6684ff1478d3b0"
I20240501 20:09:56.095033 14932 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:56.095059 14543 consensus_queue.cc:260] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:56.095609 14543 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:56.097018 14932 raft_consensus.cc:2454] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 2.
I20240501 20:09:56.097585 14433 leader_election.cc:302] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ad649017804e6faf6684ff1478d3b0, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:56.097885 15357 raft_consensus.cc:2781] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:56.098021 15357 raft_consensus.cc:686] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Becoming Leader. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:56.098409 14464 leader_election.cc:288] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 2 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133:46093)
I20240501 20:09:56.098273 15357 consensus_queue.cc:237] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:56.100021 14543 raft_consensus.cc:2454] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 2.
I20240501 20:09:56.101740 14368 catalog_manager.cc:5484] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: term changed from 1 to 2, leader changed from 3da749f295434bceb714fb91c9598d45 (127.10.219.130) to 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129). New cstate: current_term: 2 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:56.108038 14464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44444
I20240501 20:09:56.108323 14464 raft_consensus.cc:595] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Received request to transfer leadership
I20240501 20:09:56.114621 14464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44444
I20240501 20:09:56.114915 14464 raft_consensus.cc:595] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Received request to transfer leadership
I20240501 20:09:56.116371 14464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44444
I20240501 20:09:56.116685 14464 raft_consensus.cc:595] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Received request to transfer leadership
W20240501 20:09:56.117309 14416 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 9784e2a5d4324c63b42d41fe0fde222b already in progress
W20240501 20:09:56.117408 14416 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 9784e2a5d4324c63b42d41fe0fde222b already in progress
I20240501 20:09:56.297791 15340 raft_consensus.cc:1050] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa: attempting to promote NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 to VOTER
I20240501 20:09:56.298609 15340 consensus_queue.cc:237] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:56.310561 14464 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Refusing update from remote peer 12317e44612647c9bf2e942d55a89bfa: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 20:09:56.310875 15006 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEARNER]: Refusing update from remote peer 12317e44612647c9bf2e942d55a89bfa: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 20:09:56.310606 14544 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer 12317e44612647c9bf2e942d55a89bfa: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 20:09:56.311633 15369 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 20:09:56.313915 15371 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 20:09:56.314229 15340 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 20:09:56.323930 15368 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.324726 14464 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.324767 14543 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.330885 14368 catalog_manager.cc:5484] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa reported cstate change: config changed from index 30 to 31, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "12317e44612647c9bf2e942d55a89bfa" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:56.335754 15006 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.342185 14857 consensus_queue.cc:237] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:56.345865 14464 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Refusing update from remote peer 12317e44612647c9bf2e942d55a89bfa: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 20:09:56.346704 14617 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 20:09:56.347028 15368 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 20:09:56.347318 15358 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:56.348820 15378 raft_consensus.cc:1050] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: attempting to promote NON_VOTER 62ad649017804e6faf6684ff1478d3b0 to VOTER
I20240501 20:09:56.349687 15378 consensus_queue.cc:237] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } }
I20240501 20:09:56.350924 15372 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.351660 14465 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.358186 15004 raft_consensus.cc:1261] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974 [term 2 FOLLOWER]: Refusing update from remote peer 12317e44612647c9bf2e942d55a89bfa: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 20:09:56.359891 14364 catalog_manager.cc:5484] T 85e1a32af97d4a238310403e907092e0 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: config changed from index 31 to 32, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New cstate: current_term: 2 leader_uuid: "12317e44612647c9bf2e942d55a89bfa" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.360741 15317 raft_consensus.cc:979] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45: : Instructing follower d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:56.360882 15317 raft_consensus.cc:1067] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 2 LEADER]: Signalling peer d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:56.362039 14615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:38276
I20240501 20:09:56.362228 14615 raft_consensus.cc:482] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:56.362356 14615 raft_consensus.cc:3037] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:56.364642 15371 consensus_queue.cc:1035] T 85e1a32af97d4a238310403e907092e0 P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 20:09:56.365285 14615 raft_consensus.cc:504] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:56.366848 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:56.366806 15004 raft_consensus.cc:2932] T 85e1a32af97d4a238310403e907092e0 P d1eb1382cb8f47b7aef9d61088b20974 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.367239 14615 leader_election.cc:288] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 3 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:56.367161 14617 raft_consensus.cc:1226] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Rejecting Update request from peer 3da749f295434bceb714fb91c9598d45 for earlier term 2. Current term is 3. Ops: []
I20240501 20:09:56.368055 15358 consensus_queue.cc:1046] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: INVALID_TERM, Last received: 2.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:56.369429 14465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3921d54966469c8de55002d0f93a10" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:56.369652 14465 raft_consensus.cc:3037] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:56.370383 14413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 85e1a32af97d4a238310403e907092e0 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:56.373152 14465 raft_consensus.cc:2454] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 3.
I20240501 20:09:56.374516 14583 leader_election.cc:302] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c, d66b9c8dab924baa8463d26f5f25b397; no voters:
I20240501 20:09:56.374848 14544 raft_consensus.cc:3032] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:09:56.374976 14544 raft_consensus.cc:729] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:56.375066 14857 raft_consensus.cc:3037] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa [term 1 LEARNER]: Advancing to term 2
I20240501 20:09:56.375295 14544 consensus_queue.cc:260] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:56.375816 14544 raft_consensus.cc:3037] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:56.378860 15356 raft_consensus.cc:2781] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:09:56.379101 15356 raft_consensus.cc:686] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 3 LEADER]: Becoming Leader. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:56.379411 14544 raft_consensus.cc:2454] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 3.
I20240501 20:09:56.379511 15356 consensus_queue.cc:237] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:56.384178 14932 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0 [term 2 LEARNER]: Refusing update from remote peer d1eb1382cb8f47b7aef9d61088b20974: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 20:09:56.384196 14617 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Refusing update from remote peer d1eb1382cb8f47b7aef9d61088b20974: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 20:09:56.384850 15376 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 20:09:56.385303 15338 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 20:09:56.385612 14533 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e1a32af97d4a238310403e907092e0 with delete_type TABLET_DATA_TOMBSTONED (TS 3da749f295434bceb714fb91c9598d45 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:35656
I20240501 20:09:56.388865 14359 catalog_manager.cc:5484] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: term changed from 2 to 3, leader changed from 3da749f295434bceb714fb91c9598d45 (127.10.219.130) to d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131). New cstate: current_term: 3 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:56.390949 14857 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa [term 2 LEARNER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 20:09:56.391649 14464 raft_consensus.cc:1226] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Rejecting Update request from peer 3da749f295434bceb714fb91c9598d45 for earlier term 2. Current term is 3. Ops: []
I20240501 20:09:56.393854 14544 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer d1eb1382cb8f47b7aef9d61088b20974: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 20:09:56.396345 15378 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 20:09:56.399401 15382 tablet_replica.cc:330] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45: stopping tablet replica
I20240501 20:09:56.399719 15382 raft_consensus.cc:2229] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:56.400362 15382 raft_consensus.cc:2258] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:56.401363 15382 ts_tablet_manager.cc:1875] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:56.410442 15338 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:56.413240 14543 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:56.414873 14932 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:56.415256 14359 catalog_manager.cc:5484] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 reported cstate change: config changed from index 30 to 31, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d1eb1382cb8f47b7aef9d61088b20974" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:56.419584 14617 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:56.421762 15004 consensus_queue.cc:237] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } }
I20240501 20:09:56.425397 14543 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer d1eb1382cb8f47b7aef9d61088b20974: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 20:09:56.426044 15376 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 20:09:56.428568 14932 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0 [term 2 FOLLOWER]: Refusing update from remote peer d1eb1382cb8f47b7aef9d61088b20974: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 20:09:56.429534 15376 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 20:09:56.432042 15377 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:56.432688 14932 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:56.436869 15382 ts_tablet_manager.cc:1888] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240501 20:09:56.437121 15382 log.cc:1196] T 85e1a32af97d4a238310403e907092e0 P 3da749f295434bceb714fb91c9598d45: Deleting WAL directory at /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-1-root/wals/85e1a32af97d4a238310403e907092e0
I20240501 20:09:56.438531 14368 catalog_manager.cc:5484] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0 reported cstate change: config changed from index 31 to 32, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New cstate: current_term: 2 leader_uuid: "d1eb1382cb8f47b7aef9d61088b20974" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:56.439723 14413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 45bd961f0ac04a4a812a818a864c902f with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:56.440516 14355 catalog_manager.cc:4830] TS 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971): tablet 85e1a32af97d4a238310403e907092e0 (table test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]) successfully deleted
I20240501 20:09:56.443306 14607 tablet_service.cc:1514] Processing DeleteTablet for tablet 45bd961f0ac04a4a812a818a864c902f with delete_type TABLET_DATA_TOMBSTONED (TS d66b9c8dab924baa8463d26f5f25b397 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:38258
I20240501 20:09:56.445050 14543 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } }
I20240501 20:09:56.457162 15357 raft_consensus.cc:1050] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c: attempting to promote NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 to VOTER
I20240501 20:09:56.458397 15357 consensus_queue.cc:237] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 24, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:56.459075 15386 tablet_replica.cc:330] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397: stopping tablet replica
I20240501 20:09:56.459331 15386 raft_consensus.cc:2229] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:56.459666 15386 raft_consensus.cc:2258] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:56.460798 15386 ts_tablet_manager.cc:1875] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:56.468593 15004 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEARNER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 20:09:56.469357 15386 ts_tablet_manager.cc:1888] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240501 20:09:56.469502 15386 log.cc:1196] T 45bd961f0ac04a4a812a818a864c902f P d66b9c8dab924baa8463d26f5f25b397: Deleting WAL directory at /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-2-root/wals/45bd961f0ac04a4a812a818a864c902f
I20240501 20:09:56.469497 15232 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:56.470387 14353 catalog_manager.cc:4830] TS d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403): tablet 45bd961f0ac04a4a812a818a864c902f (table test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]) successfully deleted
I20240501 20:09:56.471107 14617 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 20:09:56.471623 14543 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 20:09:56.473001 15232 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:56.473338 15388 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:56.477672 15357 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.479677 14543 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.480005 14617 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.483781 14615 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 5 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
I20240501 20:09:56.484530 15388 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:56.484974 15004 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.489354 14361 catalog_manager.cc:5484] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index 26 to 27, d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.498558 14465 consensus_queue.cc:237] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } }
I20240501 20:09:56.501433 14543 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 20:09:56.502298 15388 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:56.503033 15004 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 20:09:56.503568 15388 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.002s
I20240501 20:09:56.505787 15232 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.508219 15004 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.510895 15388 raft_consensus.cc:979] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c: : Instructing follower d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:56.508072 14543 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.513423 15357 raft_consensus.cc:1067] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 5 LEADER]: Signalling peer d66b9c8dab924baa8463d26f5f25b397 to start an election
I20240501 20:09:56.515812 14617 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
from {username='slave'} at 127.0.0.1:38264
I20240501 20:09:56.521345 14368 catalog_manager.cc:5484] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974 reported cstate change: config changed from index 27 to 28, VOTER d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) evicted. New cstate: current_term: 2 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } }
I20240501 20:09:56.522974 14617 raft_consensus.cc:482] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:56.523124 14617 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:09:56.523224 14857 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 4 LEARNER]: Advancing to term 5
I20240501 20:09:56.523597 14543 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 5 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
I20240501 20:09:56.524178 15393 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:56.526470 14857 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 5 LEARNER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
I20240501 20:09:56.527542 14617 raft_consensus.cc:504] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:56.527947 14607 tablet_service.cc:1514] Processing DeleteTablet for tablet f729e4d0e8dd4afbac82612b41d1eb4a with delete_type TABLET_DATA_TOMBSTONED (TS d66b9c8dab924baa8463d26f5f25b397 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:38258
I20240501 20:09:56.528316 15386 tablet_replica.cc:330] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397: stopping tablet replica
I20240501 20:09:56.528542 15386 raft_consensus.cc:2229] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:56.528910 15386 raft_consensus.cc:2258] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:56.529389 14464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:56.529631 14464 raft_consensus.cc:3032] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:09:56.529716 14464 raft_consensus.cc:729] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:56.530185 15388 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:56.530412 15386 ts_tablet_manager.cc:1875] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:56.530838 14464 consensus_queue.cc:260] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 5.29, Last appended by leader: 29, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:56.531208 14464 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:09:56.530478 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "d66b9c8dab924baa8463d26f5f25b397" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:56.533986 14464 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66b9c8dab924baa8463d26f5f25b397 in term 6.
I20240501 20:09:56.534731 14583 leader_election.cc:302] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d368a895e2a48c8a17b384c0739e47c, d66b9c8dab924baa8463d26f5f25b397; no voters: 3da749f295434bceb714fb91c9598d45
I20240501 20:09:56.535089 15356 raft_consensus.cc:2781] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:09:56.535228 15356 raft_consensus.cc:686] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 6 LEADER]: Becoming Leader. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:56.535538 15356 consensus_queue.cc:237] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 5.29, Last appended by leader: 29, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:56.539625 14617 leader_election.cc:288] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [CANDIDATE]: Term 6 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:56.539367 14368 catalog_manager.cc:5484] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: term changed from 5 to 6, leader changed from 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) to d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131). New cstate: current_term: 6 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:56.543015 14353 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f729e4d0e8dd4afbac82612b41d1eb4a with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:56.545008 15386 ts_tablet_manager.cc:1888] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.27
I20240501 20:09:56.545193 15386 log.cc:1196] T f729e4d0e8dd4afbac82612b41d1eb4a P d66b9c8dab924baa8463d26f5f25b397: Deleting WAL directory at /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-2-root/wals/f729e4d0e8dd4afbac82612b41d1eb4a
I20240501 20:09:56.547036 14353 catalog_manager.cc:4830] TS d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403): tablet f729e4d0e8dd4afbac82612b41d1eb4a (table test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]) successfully deleted
I20240501 20:09:56.626607 14543 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 20:09:56.627236 15393 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 20:09:56.629922 15232 raft_consensus.cc:979] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c: : Instructing follower 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:56.630105 15394 raft_consensus.cc:1067] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Signalling peer 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:56.631438 14857 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 1 LEARNER]: Advancing to term 2
I20240501 20:09:56.631392 14543 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:35668
I20240501 20:09:56.631807 14932 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 20:09:56.632112 14543 raft_consensus.cc:482] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:56.632344 14543 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:56.632946 15232 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 20:09:56.634570 14857 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 2 LEARNER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 20:09:56.635241 15232 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 20:09:56.636011 14543 raft_consensus.cc:504] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:56.639750 14543 leader_election.cc:288] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 3 election: Requested vote from peers 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765), 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133:46093)
I20240501 20:09:56.639828 14932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "62ad649017804e6faf6684ff1478d3b0"
I20240501 20:09:56.640084 14932 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:56.639823 14465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
I20240501 20:09:56.640707 14465 raft_consensus.cc:3032] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:09:56.640818 14465 raft_consensus.cc:729] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:56.641105 14465 consensus_queue.cc:260] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:56.641590 14465 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:56.642882 14932 raft_consensus.cc:2454] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 3.
I20240501 20:09:56.643325 14508 leader_election.cc:302] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 62ad649017804e6faf6684ff1478d3b0; no voters:
I20240501 20:09:56.643627 15317 raft_consensus.cc:2781] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:09:56.643764 15317 raft_consensus.cc:686] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 3 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:56.644044 15317 consensus_queue.cc:237] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:56.644892 14465 raft_consensus.cc:2454] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 3.
I20240501 20:09:56.646765 14368 catalog_manager.cc:5484] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 2 to 3, leader changed from 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 3 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:56.857332 14465 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 20:09:56.857929 15279 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:56.862818 14857 raft_consensus.cc:3037] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa [term 2 LEARNER]: Advancing to term 3
I20240501 20:09:56.863853 14543 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 20:09:56.865064 15405 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 20:09:56.866355 14857 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa [term 3 LEARNER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 20:09:56.867017 15405 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:56.871006 15356 raft_consensus.cc:1050] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397: attempting to promote NON_VOTER 12317e44612647c9bf2e942d55a89bfa to VOTER
I20240501 20:09:56.872023 15356 consensus_queue.cc:237] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 26, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:56.874131 14857 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa [term 3 LEARNER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 20:09:56.874872 15279 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:56.875144 14465 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 20:09:56.875659 15279 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:56.875869 14543 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 20:09:56.876416 15279 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:56.878805 15404 raft_consensus.cc:2932] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index 25 to 28, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.879004 14465 raft_consensus.cc:2932] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 25 to 28, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.879729 14543 raft_consensus.cc:2932] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 25 to 28, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.885414 14368 catalog_manager.cc:5484] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index 25 to 28, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.886507 14857 raft_consensus.cc:2932] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 25 to 28, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.893883 14617 consensus_queue.cc:237] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:56.895920 14465 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 20:09:56.896407 14857 raft_consensus.cc:1261] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa [term 3 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 20:09:56.896485 15356 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:56.896975 15405 consensus_queue.cc:1035] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:56.899116 15404 raft_consensus.cc:2932] T 4b3921d54966469c8de55002d0f93a10 P d66b9c8dab924baa8463d26f5f25b397 [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.899838 14857 raft_consensus.cc:2932] T 4b3921d54966469c8de55002d0f93a10 P 12317e44612647c9bf2e942d55a89bfa [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.899808 14465 raft_consensus.cc:2932] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.905133 14353 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b3921d54966469c8de55002d0f93a10 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:56.905835 14361 catalog_manager.cc:5484] T 4b3921d54966469c8de55002d0f93a10 P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: config changed from index 28 to 29, VOTER 3da749f295434bceb714fb91c9598d45 (127.10.219.130) evicted. New cstate: current_term: 3 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.909584 14533 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b3921d54966469c8de55002d0f93a10 with delete_type TABLET_DATA_TOMBSTONED (TS 3da749f295434bceb714fb91c9598d45 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:35656
I20240501 20:09:56.910041 15382 tablet_replica.cc:330] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45: stopping tablet replica
I20240501 20:09:56.910243 15382 raft_consensus.cc:2229] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:56.910598 15382 raft_consensus.cc:2258] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:56.911419 15382 ts_tablet_manager.cc:1875] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:56.916556 15382 ts_tablet_manager.cc:1888] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.28
I20240501 20:09:56.916745 15382 log.cc:1196] T 4b3921d54966469c8de55002d0f93a10 P 3da749f295434bceb714fb91c9598d45: Deleting WAL directory at /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-1-root/wals/4b3921d54966469c8de55002d0f93a10
I20240501 20:09:56.917534 14355 catalog_manager.cc:4830] TS 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971): tablet 4b3921d54966469c8de55002d0f93a10 (table test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]) successfully deleted
I20240501 20:09:56.965171 14465 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 6 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 5 index: 29. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20240501 20:09:56.966325 15356 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 20:09:56.970337 14543 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:09:56.970566 14857 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 5 LEARNER]: Advancing to term 6
I20240501 20:09:56.974411 14543 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 6 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 5 index: 29. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20240501 20:09:56.975108 15404 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 20:09:56.975418 14857 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 6 LEARNER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 5 index: 29. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20240501 20:09:56.976089 15356 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 20:09:56.978627 15404 raft_consensus.cc:1050] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397: attempting to promote NON_VOTER 12317e44612647c9bf2e942d55a89bfa to VOTER
I20240501 20:09:56.979687 15404 consensus_queue.cc:237] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 6.30, Last appended by leader: 29, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:56.982357 14543 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 6 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 6 index: 31. (index mismatch)
I20240501 20:09:56.982623 14465 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 6 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 6 index: 31. (index mismatch)
I20240501 20:09:56.983137 14857 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 6 LEARNER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 6 index: 31. (index mismatch)
I20240501 20:09:56.983067 15405 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 20:09:56.983454 15404 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 20:09:56.983779 15405 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 20:09:56.986076 15404 raft_consensus.cc:2932] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 6 LEADER]: Committing config change with OpId 6.31: config changed from index 27 to 31, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.986819 14465 raft_consensus.cc:2932] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 6 FOLLOWER]: Committing config change with OpId 6.31: config changed from index 27 to 31, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.987534 14857 raft_consensus.cc:2932] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 6 FOLLOWER]: Committing config change with OpId 6.31: config changed from index 27 to 31, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.993145 14364 catalog_manager.cc:5484] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: config changed from index 27 to 31, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:56.992733 14543 raft_consensus.cc:2932] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 6 FOLLOWER]: Committing config change with OpId 6.31: config changed from index 27 to 31, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:56.999574 14617 consensus_queue.cc:237] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 6.31, Last appended by leader: 29, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:57.001591 14857 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 6 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 6 index: 31. Preceding OpId from leader: term: 6 index: 32. (index mismatch)
I20240501 20:09:57.002240 15404 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 20:09:57.002669 14543 raft_consensus.cc:1261] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 6 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 6 index: 31. Preceding OpId from leader: term: 6 index: 32. (index mismatch)
I20240501 20:09:57.003319 15404 consensus_queue.cc:1035] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 20:09:57.004598 15356 raft_consensus.cc:2932] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 6 LEADER]: Committing config change with OpId 6.32: config changed from index 31 to 32, VOTER 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.005308 14857 raft_consensus.cc:2932] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 6 FOLLOWER]: Committing config change with OpId 6.32: config changed from index 31 to 32, VOTER 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.007758 14543 raft_consensus.cc:2932] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 6 FOLLOWER]: Committing config change with OpId 6.32: config changed from index 31 to 32, VOTER 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.010793 14353 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd5f05007317454bb4e7bdefd81ccd90 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:57.010839 14359 catalog_manager.cc:5484] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa reported cstate change: config changed from index 31 to 32, VOTER 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) evicted. New cstate: current_term: 6 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.015183 14453 tablet_service.cc:1514] Processing DeleteTablet for tablet cd5f05007317454bb4e7bdefd81ccd90 with delete_type TABLET_DATA_TOMBSTONED (TS 8d368a895e2a48c8a17b384c0739e47c not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:44444
I20240501 20:09:57.016010 15408 tablet_replica.cc:330] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c: stopping tablet replica
I20240501 20:09:57.016263 15408 raft_consensus.cc:2229] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 6 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:57.016633 15408 raft_consensus.cc:2258] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c [term 6 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:57.017493 15408 ts_tablet_manager.cc:1875] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:57.022665 15408 ts_tablet_manager.cc:1888] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.31
I20240501 20:09:57.022854 15408 log.cc:1196] T cd5f05007317454bb4e7bdefd81ccd90 P 8d368a895e2a48c8a17b384c0739e47c: Deleting WAL directory at /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-0-root/wals/cd5f05007317454bb4e7bdefd81ccd90
I20240501 20:09:57.023828 14353 catalog_manager.cc:4830] TS 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765): tablet cd5f05007317454bb4e7bdefd81ccd90 (table test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]) successfully deleted
I20240501 20:09:57.096064 14857 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 2 LEARNER]: Advancing to term 3
I20240501 20:09:57.099087 14857 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 3 LEARNER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 20:09:57.099759 15317 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 20:09:57.102113 15317 raft_consensus.cc:1050] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: attempting to promote NON_VOTER 12317e44612647c9bf2e942d55a89bfa to VOTER
W20240501 20:09:57.102284 15317 raft_consensus.cc:1054] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: Unable to promote non-voter 12317e44612647c9bf2e942d55a89bfa: Illegal state: Leader has not yet committed an operation in its own term
I20240501 20:09:57.202838 14932 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 20:09:57.203485 15317 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 20:09:57.207587 14465 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 20:09:57.208324 15409 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 20:09:57.209172 15317 raft_consensus.cc:1050] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: attempting to promote NON_VOTER 12317e44612647c9bf2e942d55a89bfa to VOTER
I20240501 20:09:57.211478 15317 consensus_queue.cc:237] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:57.214157 14465 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 20:09:57.214188 14932 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 20:09:57.214852 15411 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 20:09:57.215317 15409 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 20:09:57.215507 14857 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 3 LEARNER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 20:09:57.216176 15409 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 20:09:57.217949 15411 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.218927 14857 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.218959 14465 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.223436 14932 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.223969 14368 catalog_manager.cc:5484] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa reported cstate change: config changed from index 29 to 32, 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.231067 14543 consensus_queue.cc:237] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:57.251834 14932 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 20:09:57.251863 14857 raft_consensus.cc:1261] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 3 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 20:09:57.252471 15409 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 20:09:57.252851 15317 consensus_queue.cc:1035] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 20:09:57.255136 15409 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.255795 14932 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.256239 14857 raft_consensus.cc:2932] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.260999 14361 catalog_manager.cc:5484] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa reported cstate change: config changed from index 32 to 33, VOTER 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129) evicted. New cstate: current_term: 3 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } } }
I20240501 20:09:57.262180 14355 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9784e2a5d4324c63b42d41fe0fde222b with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:57.264556 14453 tablet_service.cc:1514] Processing DeleteTablet for tablet 9784e2a5d4324c63b42d41fe0fde222b with delete_type TABLET_DATA_TOMBSTONED (TS 8d368a895e2a48c8a17b384c0739e47c not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:44444
I20240501 20:09:57.264938 15408 tablet_replica.cc:330] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c: stopping tablet replica
I20240501 20:09:57.265118 15408 raft_consensus.cc:2229] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:57.265319 15408 raft_consensus.cc:2258] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:57.266095 15408 ts_tablet_manager.cc:1875] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:57.270889 15408 ts_tablet_manager.cc:1888] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20240501 20:09:57.271039 15408 log.cc:1196] T 9784e2a5d4324c63b42d41fe0fde222b P 8d368a895e2a48c8a17b384c0739e47c: Deleting WAL directory at /tmp/dist-test-taskDkxXIF/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714594084913313-11118-0/minicluster-data/ts-0-root/wals/9784e2a5d4324c63b42d41fe0fde222b
I20240501 20:09:57.271783 14353 catalog_manager.cc:4830] TS 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765): tablet 9784e2a5d4324c63b42d41fe0fde222b (table test-workload [id=f32f5bf6de9d43d58bac8dd58a7803a9]) successfully deleted
I20240501 20:09:57.551463 14617 consensus_queue.cc:237] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:57.554265 14465 raft_consensus.cc:1261] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 20:09:57.554486 14543 raft_consensus.cc:1261] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer d66b9c8dab924baa8463d26f5f25b397: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 20:09:57.555014 15404 consensus_queue.cc:1035] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:57.555351 15415 consensus_queue.cc:1035] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 20:09:57.558450 15414 raft_consensus.cc:2932] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
W20240501 20:09:57.560618 14582 consensus_peers.cc:485] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 -> Peer 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133:46093): Couldn't send request to peer 62ad649017804e6faf6684ff1478d3b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4ddb456073f4c4aa53517b561c948af. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:57.560410 14543 raft_consensus.cc:2932] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:57.560410 14465 raft_consensus.cc:2932] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:57.564688 14464 consensus_queue.cc:237] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:57.564688 14359 catalog_manager.cc:5484] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: config changed from index -1 to 26, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New cstate: current_term: 2 leader_uuid: "d66b9c8dab924baa8463d26f5f25b397" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:57.568764 14433 consensus_peers.cc:485] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c -> Peer 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133:46093): Couldn't send request to peer 62ad649017804e6faf6684ff1478d3b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f729e4d0e8dd4afbac82612b41d1eb4a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:57.569052 15004 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 20:09:57.569685 15232 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:57.575214 15232 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:57.575767 15004 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P d1eb1382cb8f47b7aef9d61088b20974 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:57.577667 14543 raft_consensus.cc:1261] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 20:09:57.578361 15421 consensus_queue.cc:1035] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:57.580410 14543 raft_consensus.cc:2932] T f729e4d0e8dd4afbac82612b41d1eb4a P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } } }
I20240501 20:09:57.581602 14361 catalog_manager.cc:5484] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: config changed from index 28 to 29, NON_VOTER 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133) added. New cstate: current_term: 2 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:57.581789 14544 consensus_queue.cc:237] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 4.27, Last appended by leader: 26, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:57.591871 14464 raft_consensus.cc:1261] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 4 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 4 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240501 20:09:57.591871 14617 raft_consensus.cc:1261] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 4 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 4 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240501 20:09:57.592566 15412 consensus_queue.cc:1035] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240501 20:09:57.592914 14509 consensus_peers.cc:485] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 -> Peer d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134:46221): Couldn't send request to peer d1eb1382cb8f47b7aef9d61088b20974. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bf8ec6baf414c598a80753fe882d9a0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:57.592938 15409 consensus_queue.cc:1035] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:57.596532 15412 raft_consensus.cc:2932] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 [term 4 LEADER]: Committing config change with OpId 4.28: config changed from index -1 to 28, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:57.597326 14464 raft_consensus.cc:2932] T 8bf8ec6baf414c598a80753fe882d9a0 P 8d368a895e2a48c8a17b384c0739e47c [term 4 FOLLOWER]: Committing config change with OpId 4.28: config changed from index -1 to 28, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:57.598649 14617 raft_consensus.cc:2932] T 8bf8ec6baf414c598a80753fe882d9a0 P d66b9c8dab924baa8463d26f5f25b397 [term 4 FOLLOWER]: Committing config change with OpId 4.28: config changed from index -1 to 28, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:57.601558 14359 catalog_manager.cc:5484] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index -1 to 28, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New cstate: current_term: 4 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:57.603300 14464 consensus_queue.cc:237] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:57.606257 14544 raft_consensus.cc:1261] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 20:09:57.606806 14617 raft_consensus.cc:1261] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Refusing update from remote peer 8d368a895e2a48c8a17b384c0739e47c: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 20:09:57.607429 15423 consensus_queue.cc:1035] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 20:09:57.607890 15421 consensus_queue.cc:1035] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240501 20:09:57.609336 14434 consensus_peers.cc:485] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c -> Peer 12317e44612647c9bf2e942d55a89bfa (127.10.219.132:44999): Couldn't send request to peer 12317e44612647c9bf2e942d55a89bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92c570e28c114771a13fcac4fa94af43. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:57.611166 15421 raft_consensus.cc:2932] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:57.611934 14544 raft_consensus.cc:2932] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:57.616286 14617 raft_consensus.cc:2932] T 92c570e28c114771a13fcac4fa94af43 P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:57.616858 15004 consensus_queue.cc:237] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } }
I20240501 20:09:57.617120 14368 catalog_manager.cc:5484] T 92c570e28c114771a13fcac4fa94af43 P 3da749f295434bceb714fb91c9598d45 reported cstate change: config changed from index -1 to 28, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New cstate: current_term: 3 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:57.627761 14932 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0 [term 2 FOLLOWER]: Refusing update from remote peer d1eb1382cb8f47b7aef9d61088b20974: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
W20240501 20:09:57.628018 14970 consensus_peers.cc:485] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 -> Peer 12317e44612647c9bf2e942d55a89bfa (127.10.219.132:44999): Couldn't send request to peer 12317e44612647c9bf2e942d55a89bfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45bd961f0ac04a4a812a818a864c902f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:57.628448 15433 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 20:09:57.629570 14544 raft_consensus.cc:1261] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Refusing update from remote peer d1eb1382cb8f47b7aef9d61088b20974: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 20:09:57.630097 15433 consensus_queue.cc:1035] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 20:09:57.631561 15377 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:57.634752 14544 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:57.635143 14932 raft_consensus.cc:2932] T 45bd961f0ac04a4a812a818a864c902f P 62ad649017804e6faf6684ff1478d3b0 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } } }
I20240501 20:09:57.637238 14368 catalog_manager.cc:5484] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974 reported cstate change: config changed from index 32 to 33, NON_VOTER 12317e44612647c9bf2e942d55a89bfa (127.10.219.132) added. New cstate: current_term: 2 leader_uuid: "d1eb1382cb8f47b7aef9d61088b20974" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: NON_VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:57.640509 14543 consensus_queue.cc:237] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 5.28, Last appended by leader: 27, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } }
I20240501 20:09:57.643242 14464 raft_consensus.cc:1261] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 5 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 5 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
I20240501 20:09:57.643954 15411 consensus_queue.cc:1035] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:57.644135 14617 raft_consensus.cc:1261] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 5 FOLLOWER]: Refusing update from remote peer 3da749f295434bceb714fb91c9598d45: Log matching property violated. Preceding OpId in replica: term: 5 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
W20240501 20:09:57.644363 14509 consensus_peers.cc:485] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 -> Peer d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134:46221): Couldn't send request to peer d1eb1382cb8f47b7aef9d61088b20974. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b90dffdaa47425883abd651fe391006. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:57.644682 15411 consensus_queue.cc:1035] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 20:09:57.646907 15409 raft_consensus.cc:2932] T 7b90dffdaa47425883abd651fe391006 P 3da749f295434bceb714fb91c9598d45 [term 5 LEADER]: Committing config change with OpId 5.29: config changed from index -1 to 29, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:57.647528 14464 raft_consensus.cc:2932] T 7b90dffdaa47425883abd651fe391006 P 8d368a895e2a48c8a17b384c0739e47c [term 5 FOLLOWER]: Committing config change with OpId 5.29: config changed from index -1 to 29, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:57.647958 14617 raft_consensus.cc:2932] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 [term 5 FOLLOWER]: Committing config change with OpId 5.29: config changed from index -1 to 29, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:57.652922 14361 catalog_manager.cc:5484] T 7b90dffdaa47425883abd651fe391006 P d66b9c8dab924baa8463d26f5f25b397 reported cstate change: config changed from index -1 to 29, NON_VOTER d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134) added. New cstate: current_term: 5 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } attrs { replace: true } } peers { permanent_uuid: "d1eb1382cb8f47b7aef9d61088b20974" member_type: NON_VOTER last_known_addr { host: "127.10.219.134" port: 46221 } attrs { promote: true } } }
I20240501 20:09:57.818782 14422 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:09:57.822432 14543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
mode: GRACEFUL
new_leader_uuid: "12317e44612647c9bf2e942d55a89bfa"
from {username='slave'} at 127.0.0.1:35692
I20240501 20:09:57.822741 14543 raft_consensus.cc:595] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 3 LEADER]: Received request to transfer leadership to TS 12317e44612647c9bf2e942d55a89bfa
I20240501 20:09:57.823801 14617 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
mode: GRACEFUL
new_leader_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:34480
I20240501 20:09:57.824018 14617 raft_consensus.cc:595] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Received request to transfer leadership to TS 8d368a895e2a48c8a17b384c0739e47c
I20240501 20:09:57.825088 14617 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
mode: GRACEFUL
new_leader_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:34480
I20240501 20:09:57.825301 14617 raft_consensus.cc:595] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 6 LEADER]: Received request to transfer leadership to TS 3da749f295434bceb714fb91c9598d45
I20240501 20:09:57.825832 14422 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 20:09:57.826069 14422 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:09:57.930948 15440 ts_tablet_manager.cc:902] T b4ddb456073f4c4aa53517b561c948af P 62ad649017804e6faf6684ff1478d3b0: Initiating tablet copy from peer d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:57.932365 15440 tablet_copy_client.cc:308] T b4ddb456073f4c4aa53517b561c948af P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.131:39403
I20240501 20:09:57.933091 14629 tablet_copy_service.cc:140] P d66b9c8dab924baa8463d26f5f25b397: Received BeginTabletCopySession request for tablet b4ddb456073f4c4aa53517b561c948af from peer 62ad649017804e6faf6684ff1478d3b0 ({username='slave'} at 127.0.0.1:34486)
I20240501 20:09:57.933252 14629 tablet_copy_service.cc:161] P d66b9c8dab924baa8463d26f5f25b397: Beginning new tablet copy session on tablet b4ddb456073f4c4aa53517b561c948af from peer 62ad649017804e6faf6684ff1478d3b0 at {username='slave'} at 127.0.0.1:34486: session id = 62ad649017804e6faf6684ff1478d3b0-b4ddb456073f4c4aa53517b561c948af
I20240501 20:09:57.935631 14629 tablet_copy_source_session.cc:205] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:57.936800 15440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ddb456073f4c4aa53517b561c948af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:57.942662 15440 tablet_copy_client.cc:783] T b4ddb456073f4c4aa53517b561c948af P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:57.942965 15440 tablet_copy_client.cc:647] T b4ddb456073f4c4aa53517b561c948af P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:57.944562 15444 tablet_copy_client.cc:1002] T b4ddb456073f4c4aa53517b561c948af P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:57.947553 15445 ts_tablet_manager.cc:902] T f729e4d0e8dd4afbac82612b41d1eb4a P 62ad649017804e6faf6684ff1478d3b0: Initiating tablet copy from peer 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765)
I20240501 20:09:57.948859 15415 raft_consensus.cc:979] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397: : Instructing follower 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:57.949198 15415 raft_consensus.cc:1067] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 6 LEADER]: Signalling peer 3da749f295434bceb714fb91c9598d45 to start an election
I20240501 20:09:57.949875 14543 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90"
dest_uuid: "3da749f295434bceb714fb91c9598d45"
from {username='slave'} at 127.0.0.1:35670
I20240501 20:09:57.950109 14543 raft_consensus.cc:482] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:57.950243 14543 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:09:57.950258 15445 tablet_copy_client.cc:308] T f729e4d0e8dd4afbac82612b41d1eb4a P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.129:33765
I20240501 20:09:57.950990 14480 tablet_copy_service.cc:140] P 8d368a895e2a48c8a17b384c0739e47c: Received BeginTabletCopySession request for tablet f729e4d0e8dd4afbac82612b41d1eb4a from peer 62ad649017804e6faf6684ff1478d3b0 ({username='slave'} at 127.0.0.1:44506)
I20240501 20:09:57.951159 14480 tablet_copy_service.cc:161] P 8d368a895e2a48c8a17b384c0739e47c: Beginning new tablet copy session on tablet f729e4d0e8dd4afbac82612b41d1eb4a from peer 62ad649017804e6faf6684ff1478d3b0 at {username='slave'} at 127.0.0.1:44506: session id = 62ad649017804e6faf6684ff1478d3b0-f729e4d0e8dd4afbac82612b41d1eb4a
I20240501 20:09:57.953430 14543 raft_consensus.cc:504] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:57.954456 14543 leader_election.cc:288] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 7 election: Requested vote from peers d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403), 12317e44612647c9bf2e942d55a89bfa (127.10.219.132:44999)
I20240501 20:09:57.954522 14617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 7 candidate_status { last_received { term: 6 index: 32 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:57.954704 14617 raft_consensus.cc:3032] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:09:57.954772 14617 raft_consensus.cc:729] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:57.955044 14857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5f05007317454bb4e7bdefd81ccd90" candidate_uuid: "3da749f295434bceb714fb91c9598d45" candidate_term: 7 candidate_status { last_received { term: 6 index: 32 } } ignore_live_leader: true dest_uuid: "12317e44612647c9bf2e942d55a89bfa"
I20240501 20:09:57.955018 14617 consensus_queue.cc:260] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 6.32, Last appended by leader: 32, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:57.955215 14857 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:09:57.955374 14617 raft_consensus.cc:3037] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:09:57.957520 14857 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P 12317e44612647c9bf2e942d55a89bfa [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 7.
I20240501 20:09:57.957875 14617 raft_consensus.cc:2454] T cd5f05007317454bb4e7bdefd81ccd90 P d66b9c8dab924baa8463d26f5f25b397 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da749f295434bceb714fb91c9598d45 in term 7.
I20240501 20:09:57.958045 14509 leader_election.cc:302] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12317e44612647c9bf2e942d55a89bfa, 3da749f295434bceb714fb91c9598d45; no voters:
I20240501 20:09:57.958469 15409 raft_consensus.cc:2781] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:09:57.958648 15409 raft_consensus.cc:686] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [term 7 LEADER]: Becoming Leader. State: Replica: 3da749f295434bceb714fb91c9598d45, State: Running, Role: LEADER
I20240501 20:09:57.958930 15409 consensus_queue.cc:237] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 6.32, Last appended by leader: 32, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:57.960568 14480 tablet_copy_source_session.cc:205] T f729e4d0e8dd4afbac82612b41d1eb4a P 8d368a895e2a48c8a17b384c0739e47c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:57.961776 14361 catalog_manager.cc:5484] T cd5f05007317454bb4e7bdefd81ccd90 P 3da749f295434bceb714fb91c9598d45 reported cstate change: term changed from 6 to 7, leader changed from d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) to 3da749f295434bceb714fb91c9598d45 (127.10.219.130). New cstate: current_term: 7 leader_uuid: "3da749f295434bceb714fb91c9598d45" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:57.963222 15445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f729e4d0e8dd4afbac82612b41d1eb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:57.969307 15445 tablet_copy_client.cc:783] T f729e4d0e8dd4afbac82612b41d1eb4a P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:57.969743 15445 tablet_copy_client.cc:647] T f729e4d0e8dd4afbac82612b41d1eb4a P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:57.971400 15449 tablet_copy_client.cc:1002] T f729e4d0e8dd4afbac82612b41d1eb4a P 62ad649017804e6faf6684ff1478d3b0: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:58.041671 15404 raft_consensus.cc:979] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397: : Instructing follower 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:58.041874 15404 raft_consensus.cc:1067] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Signalling peer 8d368a895e2a48c8a17b384c0739e47c to start an election
I20240501 20:09:58.042523 14465 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
from {username='slave'} at 127.0.0.1:44462
I20240501 20:09:58.042775 14465 raft_consensus.cc:482] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:58.042919 14465 raft_consensus.cc:3037] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:58.045469 14465 raft_consensus.cc:504] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:58.046192 14465 leader_election.cc:288] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 3 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131:39403)
I20240501 20:09:58.046715 14617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "d66b9c8dab924baa8463d26f5f25b397"
I20240501 20:09:58.046942 14617 raft_consensus.cc:3032] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:09:58.047058 14617 raft_consensus.cc:729] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d66b9c8dab924baa8463d26f5f25b397, State: Running, Role: LEADER
I20240501 20:09:58.047024 14544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af" candidate_uuid: "8d368a895e2a48c8a17b384c0739e47c" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
I20240501 20:09:58.047223 14544 raft_consensus.cc:3037] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:58.047417 14617 consensus_queue.cc:260] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:58.047951 14617 raft_consensus.cc:3037] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:09:58.049777 14544 raft_consensus.cc:2454] T b4ddb456073f4c4aa53517b561c948af P 3da749f295434bceb714fb91c9598d45 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 3.
I20240501 20:09:58.050204 14436 leader_election.cc:302] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da749f295434bceb714fb91c9598d45, 8d368a895e2a48c8a17b384c0739e47c; no voters:
I20240501 20:09:58.050580 15423 raft_consensus.cc:2781] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:09:58.050755 15423 raft_consensus.cc:686] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 3 LEADER]: Becoming Leader. State: Replica: 8d368a895e2a48c8a17b384c0739e47c, State: Running, Role: LEADER
I20240501 20:09:58.050993 15423 consensus_queue.cc:237] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } }
I20240501 20:09:58.052824 14617 raft_consensus.cc:2454] T b4ddb456073f4c4aa53517b561c948af P d66b9c8dab924baa8463d26f5f25b397 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d368a895e2a48c8a17b384c0739e47c in term 3.
I20240501 20:09:58.053881 14361 catalog_manager.cc:5484] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c reported cstate change: term changed from 2 to 3, leader changed from d66b9c8dab924baa8463d26f5f25b397 (127.10.219.131) to 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129). New cstate: current_term: 3 leader_uuid: "8d368a895e2a48c8a17b384c0739e47c" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8d368a895e2a48c8a17b384c0739e47c" member_type: VOTER last_known_addr { host: "127.10.219.129" port: 33765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66b9c8dab924baa8463d26f5f25b397" member_type: VOTER last_known_addr { host: "127.10.219.131" port: 39403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: NON_VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:58.063011 14465 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44444
I20240501 20:09:58.063266 14465 raft_consensus.cc:595] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 3 LEADER]: Received request to transfer leadership
I20240501 20:09:58.071877 14465 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44444
I20240501 20:09:58.072101 14465 raft_consensus.cc:595] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 3 LEADER]: Received request to transfer leadership
I20240501 20:09:58.073410 14465 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4ddb456073f4c4aa53517b561c948af"
dest_uuid: "8d368a895e2a48c8a17b384c0739e47c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44444
I20240501 20:09:58.073616 14465 raft_consensus.cc:595] T b4ddb456073f4c4aa53517b561c948af P 8d368a895e2a48c8a17b384c0739e47c [term 3 LEADER]: Received request to transfer leadership
W20240501 20:09:58.074087 14416 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for b4ddb456073f4c4aa53517b561c948af already in progress
W20240501 20:09:58.074187 14416 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for b4ddb456073f4c4aa53517b561c948af already in progress
I20240501 20:09:58.104454 15451 ts_tablet_manager.cc:902] T 45bd961f0ac04a4a812a818a864c902f P 12317e44612647c9bf2e942d55a89bfa: Initiating tablet copy from peer d1eb1382cb8f47b7aef9d61088b20974 (127.10.219.134:46221)
I20240501 20:09:58.106192 15451 tablet_copy_client.cc:308] T 45bd961f0ac04a4a812a818a864c902f P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.134:46221
I20240501 20:09:58.106905 15016 tablet_copy_service.cc:140] P d1eb1382cb8f47b7aef9d61088b20974: Received BeginTabletCopySession request for tablet 45bd961f0ac04a4a812a818a864c902f from peer 12317e44612647c9bf2e942d55a89bfa ({username='slave'} at 127.0.0.1:44644)
I20240501 20:09:58.107070 15016 tablet_copy_service.cc:161] P d1eb1382cb8f47b7aef9d61088b20974: Beginning new tablet copy session on tablet 45bd961f0ac04a4a812a818a864c902f from peer 12317e44612647c9bf2e942d55a89bfa at {username='slave'} at 127.0.0.1:44644: session id = 12317e44612647c9bf2e942d55a89bfa-45bd961f0ac04a4a812a818a864c902f
I20240501 20:09:58.109773 15016 tablet_copy_source_session.cc:205] T 45bd961f0ac04a4a812a818a864c902f P d1eb1382cb8f47b7aef9d61088b20974: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:58.111187 15451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bd961f0ac04a4a812a818a864c902f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.116545 15451 tablet_copy_client.cc:783] T 45bd961f0ac04a4a812a818a864c902f P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:58.116849 15451 tablet_copy_client.cc:647] T 45bd961f0ac04a4a812a818a864c902f P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:58.118434 15455 tablet_copy_client.cc:1002] T 45bd961f0ac04a4a812a818a864c902f P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:58.185155 15456 ts_tablet_manager.cc:902] T 8bf8ec6baf414c598a80753fe882d9a0 P d1eb1382cb8f47b7aef9d61088b20974: Initiating tablet copy from peer 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
I20240501 20:09:58.187146 15456 tablet_copy_client.cc:308] T 8bf8ec6baf414c598a80753fe882d9a0 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.130:40971
I20240501 20:09:58.187809 14555 tablet_copy_service.cc:140] P 3da749f295434bceb714fb91c9598d45: Received BeginTabletCopySession request for tablet 8bf8ec6baf414c598a80753fe882d9a0 from peer d1eb1382cb8f47b7aef9d61088b20974 ({username='slave'} at 127.0.0.1:35720)
I20240501 20:09:58.187986 14555 tablet_copy_service.cc:161] P 3da749f295434bceb714fb91c9598d45: Beginning new tablet copy session on tablet 8bf8ec6baf414c598a80753fe882d9a0 from peer d1eb1382cb8f47b7aef9d61088b20974 at {username='slave'} at 127.0.0.1:35720: session id = d1eb1382cb8f47b7aef9d61088b20974-8bf8ec6baf414c598a80753fe882d9a0
I20240501 20:09:58.190997 14555 tablet_copy_source_session.cc:205] T 8bf8ec6baf414c598a80753fe882d9a0 P 3da749f295434bceb714fb91c9598d45: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:58.192301 15456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf8ec6baf414c598a80753fe882d9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:58.198670 15456 tablet_copy_client.cc:783] T 8bf8ec6baf414c598a80753fe882d9a0 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:58.198984 15456 tablet_copy_client.cc:647] T 8bf8ec6baf414c598a80753fe882d9a0 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:58.200543 15460 tablet_copy_client.cc:1002] T 8bf8ec6baf414c598a80753fe882d9a0 P d1eb1382cb8f47b7aef9d61088b20974: tablet copy: Injecting latency into file download: 2000
I20240501 20:09:58.206580 15461 ts_tablet_manager.cc:902] T 92c570e28c114771a13fcac4fa94af43 P 12317e44612647c9bf2e942d55a89bfa: Initiating tablet copy from peer 8d368a895e2a48c8a17b384c0739e47c (127.10.219.129:33765)
I20240501 20:09:58.208192 15461 tablet_copy_client.cc:308] T 92c570e28c114771a13fcac4fa94af43 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Beginning tablet copy session from remote peer at address 127.10.219.129:33765
I20240501 20:09:58.208940 14480 tablet_copy_service.cc:140] P 8d368a895e2a48c8a17b384c0739e47c: Received BeginTabletCopySession request for tablet 92c570e28c114771a13fcac4fa94af43 from peer 12317e44612647c9bf2e942d55a89bfa ({username='slave'} at 127.0.0.1:44522)
I20240501 20:09:58.209125 14480 tablet_copy_service.cc:161] P 8d368a895e2a48c8a17b384c0739e47c: Beginning new tablet copy session on tablet 92c570e28c114771a13fcac4fa94af43 from peer 12317e44612647c9bf2e942d55a89bfa at {username='slave'} at 127.0.0.1:44522: session id = 12317e44612647c9bf2e942d55a89bfa-92c570e28c114771a13fcac4fa94af43
I20240501 20:09:58.210404 15466 ts_tablet_manager.cc:902] T 7b90dffdaa47425883abd651fe391006 P d1eb1382cb8f47b7aef9d61088b20974: Initiating tablet copy from peer 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971)
F20240501 20:09:58.210876 15466 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-45bd961f0ac04a4a812a818a864c902f.queue_time_us, tablet-download-pool-45bd961f0ac04a4a812a818a864c902f.queue_time_us) (tablet-download-pool-45bd961f0ac04a4a812a818a864c902f.run_cpu_time_us, tablet-download-pool-45bd961f0ac04a4a812a818a864c902f.run_cpu_time_us) (tablet-download-pool-45bd961f0ac04a4a812a818a864c902f.run_wall_time_us, tablet-download-pool-45bd961f0ac04a4a812a818a864c902f.run_wall_time_us) (tablet-download-pool-4b3921d54966469c8de55002d0f93a10.queue_time_us, tablet-download-pool-4b3921d54966469c8de55002d0f93a10.queue_time_us) (tablet-download-pool-4b3921d54966469c8de55002d0f93a10.run_cpu_time_us, tablet-download-pool-4b3921d54966469c8de55002d0f93a10.run_cpu_time_us) (tablet-download-pool-4b3921d54966469c8de55002d0f93a10.run_wall_time_us, tablet-download-pool-4b3921d54966469c8de55002d0f93a10.run_wall_time_us) (tablet-download-pool-6fbbd45dd08144adbbd51cad42bad500.queue_time_us, tablet-download-pool-6fbbd45dd08144adbbd51cad42bad500.queue_time_us) (tablet-download-pool-6fbbd45dd08144adbbd51cad42bad500.run_cpu_time_us, tablet-download-pool-6fbbd45dd08144adbbd51cad42bad500.run_cpu_time_us) (tablet-download-pool-6fbbd45dd08144adbbd51cad42bad500.run_wall_time_us, tablet-download-pool-6fbbd45dd08144adbbd51cad42bad500.run_wall_time_us) (tablet-download-pool-7b90dffdaa47425883abd651fe391006.queue_time_us, tablet-download-pool-7b90dffdaa47425883abd651fe391006.queue_time_us) (tablet-download-pool-7c5cf239f44e48e2a85f97db902642ac.queue_time_us, tablet-download-pool-7c5cf239f44e48e2a85f97db902642ac.queue_time_us) (tablet-download-pool-7c5cf239f44e48e2a85f97db902642ac.run_cpu_time_us, tablet-download-pool-7c5cf239f44e48e2a85f97db902642ac.run_cpu_time_us) (tablet-download-pool-7c5cf239f44e48e2a85f97db902642ac.run_wall_time_us, tablet-download-pool-7c5cf239f44e48e2a85f97db902642ac.run_wall_time_us) (tablet-download-pool-85e1a32af97d4a238310403e907092e0.queue_time_us, tablet-download-pool-85e1a32af97d4a238310403e907092e0.queue_time_us) (tablet-download-pool-85e1a32af97d4a238310403e907092e0.run_cpu_time_us, tablet-download-pool-85e1a32af97d4a238310403e907092e0.run_cpu_time_us) (tablet-download-pool-85e1a32af97d4a238310403e907092e0.run_wall_time_us, tablet-download-pool-85e1a32af97d4a238310403e907092e0.run_wall_time_us) (tablet-download-pool-8bf8ec6baf414c598a80753fe882d9a0.queue_time_us, tablet-download-pool-8bf8ec6baf414c598a80753fe882d9a0.queue_time_us) (tablet-download-pool-8bf8ec6baf414c598a80753fe882d9a0.run_cpu_time_us, tablet-download-pool-8bf8ec6baf414c598a80753fe882d9a0.run_cpu_time_us) (tablet-download-pool-8bf8ec6baf414c598a80753fe882d9a0.run_wall_time_us, tablet-download-pool-8bf8ec6baf414c598a80753fe882d9a0.run_wall_time_us) (tablet-download-pool-92c570e28c114771a13fcac4fa94af43.queue_time_us, tablet-download-pool-92c570e28c114771a13fcac4fa94af43.queue_time_us) (tablet-download-pool-92c570e28c114771a13fcac4fa94af43.run_cpu_time_us, tablet-download-pool-92c570e28c114771a13fcac4fa94af43.run_cpu_time_us) (tablet-download-pool-92c570e28c114771a13fcac4fa94af43.run_wall_time_us, tablet-download-pool-92c570e28c114771a13fcac4fa94af43.run_wall_time_us) (tablet-download-pool-9784e2a5d4324c63b42d41fe0fde222b.queue_time_us, tablet-download-pool-9784e2a5d4324c63b42d41fe0fde222b.queue_time_us) (tablet-download-pool-9784e2a5d4324c63b42d41fe0fde222b.run_cpu_time_us, tablet-download-pool-9784e2a5d4324c63b42d41fe0fde222b.run_cpu_time_us) (tablet-download-pool-9784e2a5d4324c63b42d41fe0fde222b.run_wall_time_us, tablet-download-pool-9784e2a5d4324c63b42d41fe0fde222b.run_wall_time_us) (tablet-download-pool-b4ddb456073f4c4aa53517b561c948af.queue_time_us, tablet-download-pool-b4ddb456073f4c4aa53517b561c948af.queue_time_us) (tablet-download-pool-b4ddb456073f4c4aa53517b561c948af.run_cpu_time_us, tablet-download-pool-b4ddb456073f4c4aa53517b561c948af.run_cpu_time_us) (tablet-download-pool-b4ddb456073f4c4aa53517b561c948af.run_wall_time_us, tablet-download-pool-b4ddb456073f4c4aa53517b561c948af.run_wall_time_us) (tablet-download-pool-baf26068e4c246baa73afccb4e93c847.queue_time_us, tablet-download-pool-baf26068e4c246baa73afccb4e93c847.queue_time_us) (tablet-download-pool-baf26068e4c246baa73afccb4e93c847.run_cpu_time_us, tablet-download-pool-baf26068e4c246baa73afccb4e93c847.run_cpu_time_us) (tablet-download-pool-baf26068e4c246baa73afccb4e93c847.run_wall_time_us, tablet-download-pool-baf26068e4c246baa73afccb4e93c847.run_wall_time_us) (tablet-download-pool-cd5f05007317454bb4e7bdefd81ccd90.queue_time_us, tablet-download-pool-cd5f05007317454bb4e7bdefd81ccd90.queue_time_us) (tablet-download-pool-cd5f05007317454bb4e7bdefd81ccd90.run_cpu_time_us, tablet-download-pool-cd5f05007317454bb4e7bdefd81ccd90.run_cpu_time_us) (tablet-download-pool-cd5f05007317454bb4e7bdefd81ccd90.run_wall_time_us, tablet-download-pool-cd5f05007317454bb4e7bdefd81ccd90.run_wall_time_us) (tablet-download-pool-f729e4d0e8dd4afbac82612b41d1eb4a.queue_time_us, tablet-download-pool-f729e4d0e8dd4afbac82612b41d1eb4a.queue_time_us) (tablet-download-pool-f729e4d0e8dd4afbac82612b41d1eb4a.run_cpu_time_us, tablet-download-pool-f729e4d0e8dd4afbac82612b41d1eb4a.run_cpu_time_us) (tablet-download-pool-f729e4d0e8dd4afbac82612b41d1eb4a.run_wall_time_us, tablet-download-pool-f729e4d0e8dd4afbac82612b41d1eb4a.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240501 20:09:58.212540 14480 tablet_copy_source_session.cc:205] T 92c570e28c114771a13fcac4fa94af43 P 8d368a895e2a48c8a17b384c0739e47c: Tablet Copy: opened 0 blocks and 1 log segments
*** Check failure stack trace: ***
*** Aborted at 1714594198 (unix time) try "date -d @1714594198" if you are using GNU date ***
I20240501 20:09:58.214067 15461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c570e28c114771a13fcac4fa94af43. 1 dirs total, 0 dirs full, 0 dirs failed
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800002b6e) received by PID 11118 (TID 0x7fbab5157700) from PID 11118; stack trace: ***
I20240501 20:09:58.223026 15461 tablet_copy_client.cc:783] T 92c570e28c114771a13fcac4fa94af43 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:58.223415 15461 tablet_copy_client.cc:647] T 92c570e28c114771a13fcac4fa94af43 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:58.225204 15465 tablet_copy_client.cc:1002] T 92c570e28c114771a13fcac4fa94af43 P 12317e44612647c9bf2e942d55a89bfa: tablet copy: Injecting latency into file download: 2000
@ 0x7fbb7608f980 (unknown) at ??:0
@ 0x7fbb6b3dcfb7 gsignal at ??:0
@ 0x7fbb6b3de921 abort at ??:0
@ 0x7fbb6de91dcd google::LogMessage::Fail() at ??:0
@ 0x7fbb6de95b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fbb6de917cc google::LogMessage::Flush() at ??:0
@ 0x7fbb6de92f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbb6f50e742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fbb6f4daf8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fbb6f4d4cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fbb7e9a08de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fbb7e9b8daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fbb7eae3480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fbb7eb1ba8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fbb7eafe981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fbb7eafe70e std::_Function_handler<>::_M_invoke() at ??:0
I20240501 20:09:58.315336 15358 raft_consensus.cc:979] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45: : Instructing follower 12317e44612647c9bf2e942d55a89bfa to start an election
I20240501 20:09:58.315548 15358 raft_consensus.cc:1067] T 9784e2a5d4324c63b42d41fe0fde222b P 3da749f295434bceb714fb91c9598d45 [term 3 LEADER]: Signalling peer 12317e44612647c9bf2e942d55a89bfa to start an election
I20240501 20:09:58.316188 14857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b"
dest_uuid: "12317e44612647c9bf2e942d55a89bfa"
from {username='slave'} at 127.0.0.1:36548
I20240501 20:09:58.316412 14857 raft_consensus.cc:482] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:58.316546 14857 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 3 FOLLOWER]: Advancing to term 4
@ 0x7fbb839a098d std::function<>::operator()() at ??:0
I20240501 20:09:58.319132 14857 raft_consensus.cc:504] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
I20240501 20:09:58.319937 14857 leader_election.cc:288] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [CANDIDATE]: Term 4 election: Requested vote from peers 3da749f295434bceb714fb91c9598d45 (127.10.219.130:40971), 62ad649017804e6faf6684ff1478d3b0 (127.10.219.133:46093)
I20240501 20:09:58.320421 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b" candidate_uuid: "12317e44612647c9bf2e942d55a89bfa" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "3da749f295434bceb714fb91c9598d45"
@ 0x7fbb6f4deb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbb6f4e219c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbb6f4e2019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fbb839a098d std::function<>::operator()() at ??:0
@ 0x7fbb6f4bd01e kudu::Thread::SuperviseThread() at ??:0
I20240501 20:09:58.353144 14932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9784e2a5d4324c63b42d41fe0fde222b" candidate_uuid: "12317e44612647c9bf2e942d55a89bfa" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "62ad649017804e6faf6684ff1478d3b0"
I20240501 20:09:58.353394 14932 raft_consensus.cc:3037] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:09:58.356379 14932 raft_consensus.cc:2454] T 9784e2a5d4324c63b42d41fe0fde222b P 62ad649017804e6faf6684ff1478d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12317e44612647c9bf2e942d55a89bfa in term 4.
@ 0x7fbb760846db start_thread at ??:0
I20240501 20:09:58.357024 14820 leader_election.cc:302] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12317e44612647c9bf2e942d55a89bfa, 62ad649017804e6faf6684ff1478d3b0; no voters:
I20240501 20:09:58.357739 15371 raft_consensus.cc:2781] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:09:58.359264 15371 raft_consensus.cc:686] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [term 4 LEADER]: Becoming Leader. State: Replica: 12317e44612647c9bf2e942d55a89bfa, State: Running, Role: LEADER
I20240501 20:09:58.359764 15371 consensus_queue.cc:237] T 9784e2a5d4324c63b42d41fe0fde222b P 12317e44612647c9bf2e942d55a89bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3da749f295434bceb714fb91c9598d45" member_type: VOTER last_known_addr { host: "127.10.219.130" port: 40971 } } peers { permanent_uuid: "62ad649017804e6faf6684ff1478d3b0" member_type: VOTER last_known_addr { host: "127.10.219.133" port: 46093 } attrs { promote: false } } peers { permanent_uuid: "12317e44612647c9bf2e942d55a89bfa" member_type: VOTER last_known_addr { host: "127.10.219.132" port: 44999 } attrs { promote: false } }
@ 0x7fbb6b4bf71f clone at ??:0