[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 08:11:06.794939 920 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.230.62:44059,127.0.230.61:42425,127.0.230.60:45915
I20240503 08:11:06.796623 920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:06.797796 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:06.808666 926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.809106 928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.809376 925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:06.810043 920 server_base.cc:1034] running on GCE node
I20240503 08:11:06.810811 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:06.810973 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:06.811054 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723866811034 us; error 0 us; skew 500 ppm
I20240503 08:11:06.811352 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:06.818581 920 webserver.cc:460] Webserver started at http://127.0.230.62:34827/ using document root <none> and password file <none>
I20240503 08:11:06.819329 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:06.819469 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:06.819825 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:06.823216 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-0-root/instance:
uuid: "d701dd16e12b424cafeb6c93949c3499"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-2jf9"
I20240503 08:11:06.829041 920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 08:11:06.832329 933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:06.833819 920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 08:11:06.833992 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-0-root
uuid: "d701dd16e12b424cafeb6c93949c3499"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-2jf9"
I20240503 08:11:06.834165 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:06.851616 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:06.852387 920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:06.852661 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:06.869846 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.62:44059
I20240503 08:11:06.869877 984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.62:44059 every 8 connection(s)
I20240503 08:11:06.872391 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:06.875152 987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.875811 988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:06.876207 985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:06.876403 920 server_base.cc:1034] running on GCE node
W20240503 08:11:06.877020 990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:06.877401 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:06.877481 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:06.877565 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723866877528 us; error 0 us; skew 500 ppm
I20240503 08:11:06.877717 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:06.879040 920 webserver.cc:460] Webserver started at http://127.0.230.61:46451/ using document root <none> and password file <none>
I20240503 08:11:06.879251 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:06.879348 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:06.879521 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:06.880146 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-1-root/instance:
uuid: "cd67ee33153c457a87de4e87dec40947"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-2jf9"
I20240503 08:11:06.882719 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:06.884534 995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:06.884913 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:06.885025 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-1-root
uuid: "cd67ee33153c457a87de4e87dec40947"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-2jf9"
I20240503 08:11:06.885165 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:06.887409 985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:06.903697 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:06.904345 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:06.905227 985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } has no permanent_uuid. Determining permanent_uuid...
W20240503 08:11:06.906374 937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.230.61:42425: connect: Connection refused (error 111)
W20240503 08:11:06.907927 985 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.230.61:42425: Network error: Client connection negotiation failed: client connection to 127.0.230.61:42425: connect: Connection refused (error 111)
I20240503 08:11:06.919096 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.61:42425
I20240503 08:11:06.919131 1049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.61:42425 every 8 connection(s)
I20240503 08:11:06.920456 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:11:06.920738 1050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:11:06.923424 1052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.923919 1053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.924427 1055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:06.924871 1050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:06.926995 920 server_base.cc:1034] running on GCE node
I20240503 08:11:06.927302 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:06.927423 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:06.927502 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723866927462 us; error 0 us; skew 500 ppm
I20240503 08:11:06.927677 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:06.929011 920 webserver.cc:460] Webserver started at http://127.0.230.60:45099/ using document root <none> and password file <none>
I20240503 08:11:06.929214 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:06.929314 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:06.929457 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:06.930297 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-2-root/instance:
uuid: "42501d4ad9334f36b3211d7b6f0622dd"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-2jf9"
I20240503 08:11:06.931514 1050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:06.933593 920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 08:11:06.935473 1062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:06.935868 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:11:06.935997 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-2-root
uuid: "42501d4ad9334f36b3211d7b6f0622dd"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-2jf9"
I20240503 08:11:06.936148 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:06.938702 1050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } has no permanent_uuid. Determining permanent_uuid...
W20240503 08:11:06.940661 1050 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.230.60:45915: Network error: Client connection negotiation failed: client connection to 127.0.230.60:45915: connect: Connection refused (error 111)
I20240503 08:11:06.952389 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:06.952988 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:06.957255 985 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } attempt: 1
I20240503 08:11:06.965166 985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } has no permanent_uuid. Determining permanent_uuid...
W20240503 08:11:06.967383 985 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.230.60:45915: Network error: Client connection negotiation failed: client connection to 127.0.230.60:45915: connect: Connection refused (error 111)
I20240503 08:11:06.968945 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.60:45915
I20240503 08:11:06.969007 1114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.60:45915 every 8 connection(s)
I20240503 08:11:06.969919 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:11:06.970348 1115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:06.972828 1115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:06.978775 1115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:06.983666 1115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:06.992864 1050 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } attempt: 1
I20240503 08:11:06.993726 1115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd: Bootstrap starting.
I20240503 08:11:06.998211 1115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:06.999751 1115 log.cc:824] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:07.001708 1050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947: Bootstrap starting.
I20240503 08:11:07.002890 1115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd: No bootstrap required, opened a new log
I20240503 08:11:07.003366 1050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:07.005435 1050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947: No bootstrap required, opened a new log
I20240503 08:11:07.010615 985 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } attempt: 1
I20240503 08:11:07.010566 1115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } }
I20240503 08:11:07.010568 1050 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } }
I20240503 08:11:07.011046 1115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:07.011052 1050 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:07.011217 1050 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67ee33153c457a87de4e87dec40947, State: Initialized, Role: FOLLOWER
I20240503 08:11:07.011221 1115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42501d4ad9334f36b3211d7b6f0622dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:07.012199 1115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } }
I20240503 08:11:07.012205 1050 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } }
I20240503 08:11:07.013250 1121 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } } }
I20240503 08:11:07.013576 1121 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:07.014086 1122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } } }
I20240503 08:11:07.014364 1122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:07.015012 1115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:07.015012 1050 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:07.020097 985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499: Bootstrap starting.
I20240503 08:11:07.022114 985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:07.024381 985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499: No bootstrap required, opened a new log
I20240503 08:11:07.025245 985 raft_consensus.cc:348] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } }
I20240503 08:11:07.025548 985 raft_consensus.cc:374] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:07.025684 985 raft_consensus.cc:729] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d701dd16e12b424cafeb6c93949c3499, State: Initialized, Role: FOLLOWER
I20240503 08:11:07.026062 985 consensus_queue.cc:260] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } }
W20240503 08:11:07.028247 1141 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:11:07.028406 1141 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:11:07.028900 1145 sys_catalog.cc:455] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } } }
I20240503 08:11:07.029215 1145 sys_catalog.cc:458] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:07.029323 985 sys_catalog.cc:564] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [sys.catalog]: configured and running, proceeding with master startup.
W20240503 08:11:07.031682 1147 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:11:07.031805 1147 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:11:07.034482 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240503 08:11:07.034600 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 08:11:07.035488 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:07.036060 1158 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:11:07.036164 1158 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 08:11:07.039373 1159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.047752 1160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.048612 1162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.049021 920 server_base.cc:1034] running on GCE node
I20240503 08:11:07.049315 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:07.049417 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:07.049480 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723867049460 us; error 0 us; skew 500 ppm
I20240503 08:11:07.049701 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.051175 920 webserver.cc:460] Webserver started at http://127.0.230.1:45393/ using document root <none> and password file <none>
I20240503 08:11:07.051515 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.051626 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.051800 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.052757 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-0-root/instance:
uuid: "584356bd4c4047b7b32f16de7d96d3a5"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-2jf9"
I20240503 08:11:07.056350 920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:07.058575 1167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.059015 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:11:07.059125 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-0-root
uuid: "584356bd4c4047b7b32f16de7d96d3a5"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-2jf9"
I20240503 08:11:07.059259 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:07.080909 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:07.081578 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:07.082466 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:07.083961 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:07.084074 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.084183 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:07.084260 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.103150 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.1:32909
I20240503 08:11:07.103183 1229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.1:32909 every 8 connection(s)
W20240503 08:11:07.104804 939 tablet.cc:2369] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:07.121425 1231 heartbeater.cc:347] Connected to a master server at 127.0.230.62:44059
I20240503 08:11:07.121789 1231 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.123136 1231 heartbeater.cc:511] Master 127.0.230.62:44059 requested a full tablet report, sending...
I20240503 08:11:07.126271 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:11:07.127465 950 ts_manager.cc:194] Registered new tserver with Master: 584356bd4c4047b7b32f16de7d96d3a5 (127.0.230.1:32909)
I20240503 08:11:07.130153 1230 heartbeater.cc:347] Connected to a master server at 127.0.230.60:45915
I20240503 08:11:07.130286 1230 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.131040 1230 heartbeater.cc:511] Master 127.0.230.60:45915 requested a full tablet report, sending...
I20240503 08:11:07.132380 1080 ts_manager.cc:194] Registered new tserver with Master: 584356bd4c4047b7b32f16de7d96d3a5 (127.0.230.1:32909)
I20240503 08:11:07.134718 1232 heartbeater.cc:347] Connected to a master server at 127.0.230.61:42425
I20240503 08:11:07.134882 1232 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.135483 1232 heartbeater.cc:511] Master 127.0.230.61:42425 requested a full tablet report, sending...
W20240503 08:11:07.136047 1236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.137225 1015 ts_manager.cc:194] Registered new tserver with Master: 584356bd4c4047b7b32f16de7d96d3a5 (127.0.230.1:32909)
W20240503 08:11:07.141074 1237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.149210 920 server_base.cc:1034] running on GCE node
W20240503 08:11:07.149868 1239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.150274 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:07.150346 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:07.150413 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723867150377 us; error 0 us; skew 500 ppm
I20240503 08:11:07.150579 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.152089 920 webserver.cc:460] Webserver started at http://127.0.230.2:36883/ using document root <none> and password file <none>
I20240503 08:11:07.152333 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.152436 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.152583 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.153282 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-1-root/instance:
uuid: "c779bc9db6244945af9789bd5ea66221"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-2jf9"
I20240503 08:11:07.155920 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20240503 08:11:07.158030 1244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.158399 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 08:11:07.158501 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-1-root
uuid: "c779bc9db6244945af9789bd5ea66221"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-2jf9"
I20240503 08:11:07.158638 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:07.167526 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:07.168176 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:07.168821 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:07.169834 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:07.169907 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.170004 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:07.170053 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.188849 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.2:44413
I20240503 08:11:07.188870 1306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.2:44413 every 8 connection(s)
I20240503 08:11:07.201900 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:11:07.206393 1308 heartbeater.cc:347] Connected to a master server at 127.0.230.62:44059
I20240503 08:11:07.206568 1308 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.207150 1308 heartbeater.cc:511] Master 127.0.230.62:44059 requested a full tablet report, sending...
I20240503 08:11:07.207585 1307 heartbeater.cc:347] Connected to a master server at 127.0.230.60:45915
I20240503 08:11:07.207707 1307 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.208312 1307 heartbeater.cc:511] Master 127.0.230.60:45915 requested a full tablet report, sending...
I20240503 08:11:07.209357 950 ts_manager.cc:194] Registered new tserver with Master: c779bc9db6244945af9789bd5ea66221 (127.0.230.2:44413)
I20240503 08:11:07.210763 1080 ts_manager.cc:194] Registered new tserver with Master: c779bc9db6244945af9789bd5ea66221 (127.0.230.2:44413)
I20240503 08:11:07.212215 1309 heartbeater.cc:347] Connected to a master server at 127.0.230.61:42425
I20240503 08:11:07.212338 1309 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.213140 1309 heartbeater.cc:511] Master 127.0.230.61:42425 requested a full tablet report, sending...
W20240503 08:11:07.213372 1313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.214048 1015 ts_manager.cc:194] Registered new tserver with Master: c779bc9db6244945af9789bd5ea66221 (127.0.230.2:44413)
W20240503 08:11:07.214752 1314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.218967 920 server_base.cc:1034] running on GCE node
W20240503 08:11:07.219673 1316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.220067 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:07.220155 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:07.220213 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723867220193 us; error 0 us; skew 500 ppm
I20240503 08:11:07.220386 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.221901 920 webserver.cc:460] Webserver started at http://127.0.230.3:36477/ using document root <none> and password file <none>
I20240503 08:11:07.222121 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.222203 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.222347 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.223002 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-2-root/instance:
uuid: "5b770b3463d04b75a97b046626227b73"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-2jf9"
I20240503 08:11:07.225557 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:07.227267 1321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.227687 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:07.227829 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-2-root
uuid: "5b770b3463d04b75a97b046626227b73"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-2jf9"
I20240503 08:11:07.227960 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:07.249053 1145 raft_consensus.cc:482] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:07.249347 1145 raft_consensus.cc:504] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } }
I20240503 08:11:07.251046 1145 leader_election.cc:288] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67ee33153c457a87de4e87dec40947 (127.0.230.61:42425), 42501d4ad9334f36b3211d7b6f0622dd (127.0.230.60:45915)
I20240503 08:11:07.251828 1090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d701dd16e12b424cafeb6c93949c3499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42501d4ad9334f36b3211d7b6f0622dd" is_pre_election: true
I20240503 08:11:07.252247 1090 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d701dd16e12b424cafeb6c93949c3499 in term 0.
I20240503 08:11:07.252292 1025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d701dd16e12b424cafeb6c93949c3499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67ee33153c457a87de4e87dec40947" is_pre_election: true
I20240503 08:11:07.252528 1025 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d701dd16e12b424cafeb6c93949c3499 in term 0.
I20240503 08:11:07.253029 937 leader_election.cc:302] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42501d4ad9334f36b3211d7b6f0622dd, d701dd16e12b424cafeb6c93949c3499; no voters:
I20240503 08:11:07.253417 1145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:07.253571 1145 raft_consensus.cc:482] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:07.253669 1145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:07.255419 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:07.256003 1145 raft_consensus.cc:504] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } }
I20240503 08:11:07.256181 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:07.256631 1145 leader_election.cc:288] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [CANDIDATE]: Term 1 election: Requested vote from peers cd67ee33153c457a87de4e87dec40947 (127.0.230.61:42425), 42501d4ad9334f36b3211d7b6f0622dd (127.0.230.60:45915)
I20240503 08:11:07.257118 1090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d701dd16e12b424cafeb6c93949c3499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42501d4ad9334f36b3211d7b6f0622dd"
I20240503 08:11:07.257298 1090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:07.257431 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:07.257090 1025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d701dd16e12b424cafeb6c93949c3499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67ee33153c457a87de4e87dec40947"
I20240503 08:11:07.257570 1025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:07.260617 1025 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d701dd16e12b424cafeb6c93949c3499 in term 1.
I20240503 08:11:07.260682 1090 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d701dd16e12b424cafeb6c93949c3499 in term 1.
I20240503 08:11:07.261056 937 leader_election.cc:302] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42501d4ad9334f36b3211d7b6f0622dd, d701dd16e12b424cafeb6c93949c3499; no voters:
I20240503 08:11:07.261474 1145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:07.261478 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:07.261641 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.261756 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:07.261847 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.262091 1145 raft_consensus.cc:686] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 1 LEADER]: Becoming Leader. State: Replica: d701dd16e12b424cafeb6c93949c3499, State: Running, Role: LEADER
I20240503 08:11:07.262780 1145 consensus_queue.cc:237] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } }
I20240503 08:11:07.269048 1328 sys_catalog.cc:455] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d701dd16e12b424cafeb6c93949c3499. Latest consensus state: current_term: 1 leader_uuid: "d701dd16e12b424cafeb6c93949c3499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } } }
I20240503 08:11:07.269474 1328 sys_catalog.cc:458] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:07.270577 1341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:07.273049 1341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:07.285450 1090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [term 1 FOLLOWER]: Refusing update from remote peer d701dd16e12b424cafeb6c93949c3499: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:07.286196 1025 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [term 1 FOLLOWER]: Refusing update from remote peer d701dd16e12b424cafeb6c93949c3499: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:07.287020 1328 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:07.287907 1145 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:07.306965 1122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d701dd16e12b424cafeb6c93949c3499. Latest consensus state: current_term: 1 leader_uuid: "d701dd16e12b424cafeb6c93949c3499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } } }
I20240503 08:11:07.307317 1122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:07.309725 1364 mvcc.cc:205] Tried to move back new op lower bound from 7023508960390606848 to 7023508960314036224. Current Snapshot: MvccSnapshot[applied={T|T < 7023508960390606848}]
I20240503 08:11:07.316143 1121 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d701dd16e12b424cafeb6c93949c3499. Latest consensus state: current_term: 1 leader_uuid: "d701dd16e12b424cafeb6c93949c3499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } } }
I20240503 08:11:07.316478 1121 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:07.320346 1328 sys_catalog.cc:455] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d701dd16e12b424cafeb6c93949c3499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } } }
I20240503 08:11:07.320652 1328 sys_catalog.cc:458] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:07.321058 1328 sys_catalog.cc:455] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d701dd16e12b424cafeb6c93949c3499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } } }
I20240503 08:11:07.321290 1328 sys_catalog.cc:458] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:07.322010 1122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d701dd16e12b424cafeb6c93949c3499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } } }
I20240503 08:11:07.322201 1122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:07.324025 1341 catalog_manager.cc:1347] Generated new cluster ID: b076ebadbb1c44f78731ed4c681b918e
I20240503 08:11:07.324132 1341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:07.327243 1121 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d701dd16e12b424cafeb6c93949c3499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d701dd16e12b424cafeb6c93949c3499" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 44059 } } peers { permanent_uuid: "cd67ee33153c457a87de4e87dec40947" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 42425 } } peers { permanent_uuid: "42501d4ad9334f36b3211d7b6f0622dd" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 45915 } } }
I20240503 08:11:07.327535 1121 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:07.328424 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.3:39909
I20240503 08:11:07.328465 1394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.3:39909 every 8 connection(s)
I20240503 08:11:07.351846 1395 heartbeater.cc:347] Connected to a master server at 127.0.230.60:45915
I20240503 08:11:07.352025 1395 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.352766 1395 heartbeater.cc:511] Master 127.0.230.60:45915 requested a full tablet report, sending...
I20240503 08:11:07.356696 1341 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:07.358552 1341 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:07.375237 1080 ts_manager.cc:194] Registered new tserver with Master: 5b770b3463d04b75a97b046626227b73 (127.0.230.3:39909)
I20240503 08:11:07.375813 1396 heartbeater.cc:347] Connected to a master server at 127.0.230.62:44059
I20240503 08:11:07.375972 1396 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.376612 1396 heartbeater.cc:511] Master 127.0.230.62:44059 requested a full tablet report, sending...
I20240503 08:11:07.377342 1398 heartbeater.cc:347] Connected to a master server at 127.0.230.61:42425
I20240503 08:11:07.377492 1398 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.377683 950 ts_manager.cc:194] Registered new tserver with Master: 5b770b3463d04b75a97b046626227b73 (127.0.230.3:39909)
I20240503 08:11:07.378086 1398 heartbeater.cc:511] Master 127.0.230.61:42425 requested a full tablet report, sending...
I20240503 08:11:07.379042 1015 ts_manager.cc:194] Registered new tserver with Master: 5b770b3463d04b75a97b046626227b73 (127.0.230.3:39909)
I20240503 08:11:07.379781 920 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.000311202s
I20240503 08:11:07.387267 1341 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499: Generated new TSK 0
I20240503 08:11:07.388568 1341 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:08.030300 1141 catalog_manager.cc:1259] Loaded cluster ID: b076ebadbb1c44f78731ed4c681b918e
I20240503 08:11:08.030406 1141 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd: loading cluster ID for follower catalog manager: success
I20240503 08:11:08.032912 1141 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd: acquiring CA information for follower catalog manager: success
I20240503 08:11:08.033082 1147 catalog_manager.cc:1259] Loaded cluster ID: b076ebadbb1c44f78731ed4c681b918e
I20240503 08:11:08.033180 1147 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947: loading cluster ID for follower catalog manager: success
I20240503 08:11:08.034669 1141 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:11:08.035156 1147 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947: acquiring CA information for follower catalog manager: success
I20240503 08:11:08.036140 1147 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:11:08.131724 950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41316
I20240503 08:11:08.212532 950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41332
I20240503 08:11:08.380237 950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41346
I20240503 08:11:09.034538 1157 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:09.133677 1231 heartbeater.cc:503] Master 127.0.230.62:44059 was elected leader, sending a full tablet report...
I20240503 08:11:09.213696 1308 heartbeater.cc:503] Master 127.0.230.62:44059 was elected leader, sending a full tablet report...
I20240503 08:11:09.381392 1396 heartbeater.cc:503] Master 127.0.230.62:44059 was elected leader, sending a full tablet report...
I20240503 08:11:09.539150 920 test_util.cc:268] Using random seed: 1126284086
I20240503 08:11:09.564559 950 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41354:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 08:11:09.606382 1272 tablet_service.cc:1467] Processing CreateTablet for tablet ace966a71f5e4bd0adb7e4dd209a6188 (DEFAULT_TABLE table=test-workload [id=455beb9f721940269a11d85019df6f61]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:11:09.607340 1272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace966a71f5e4bd0adb7e4dd209a6188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:09.610817 1352 tablet_service.cc:1467] Processing CreateTablet for tablet 799dcf6148774ecfb4cfc15207274bff (DEFAULT_TABLE table=test-workload [id=455beb9f721940269a11d85019df6f61]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:11:09.611265 1352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 799dcf6148774ecfb4cfc15207274bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:09.614152 1431 tablet_bootstrap.cc:492] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221: Bootstrap starting.
I20240503 08:11:09.614518 1195 tablet_service.cc:1467] Processing CreateTablet for tablet 03abec94ec6a4ecf8af1a987d115f2f1 (DEFAULT_TABLE table=test-workload [id=455beb9f721940269a11d85019df6f61]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:11:09.615147 1195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03abec94ec6a4ecf8af1a987d115f2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:09.616060 1194 tablet_service.cc:1467] Processing CreateTablet for tablet 9151fb6f49a14115b0326ec35ab0e680 (DEFAULT_TABLE table=test-workload [id=455beb9f721940269a11d85019df6f61]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:11:09.616662 1194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9151fb6f49a14115b0326ec35ab0e680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:09.617054 1431 tablet_bootstrap.cc:654] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:09.623907 1432 tablet_bootstrap.cc:492] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73: Bootstrap starting.
I20240503 08:11:09.624375 1431 tablet_bootstrap.cc:492] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221: No bootstrap required, opened a new log
I20240503 08:11:09.624639 1431 ts_tablet_manager.cc:1367] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 08:11:09.625627 1431 raft_consensus.cc:348] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c779bc9db6244945af9789bd5ea66221" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44413 } }
I20240503 08:11:09.625810 1431 raft_consensus.cc:374] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:09.625926 1431 raft_consensus.cc:729] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c779bc9db6244945af9789bd5ea66221, State: Initialized, Role: FOLLOWER
I20240503 08:11:09.626199 1431 consensus_queue.cc:260] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c779bc9db6244945af9789bd5ea66221" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44413 } }
I20240503 08:11:09.626420 1431 raft_consensus.cc:388] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:09.626458 1432 tablet_bootstrap.cc:654] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:09.626557 1431 raft_consensus.cc:482] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:09.626718 1431 raft_consensus.cc:3037] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:09.629145 1431 raft_consensus.cc:504] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c779bc9db6244945af9789bd5ea66221" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44413 } }
I20240503 08:11:09.629380 1431 leader_election.cc:302] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c779bc9db6244945af9789bd5ea66221; no voters:
I20240503 08:11:09.629865 1431 leader_election.cc:288] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:09.630043 1436 raft_consensus.cc:2781] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:09.635965 1432 tablet_bootstrap.cc:492] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73: No bootstrap required, opened a new log
I20240503 08:11:09.636184 1432 ts_tablet_manager.cc:1367] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240503 08:11:09.636611 1434 tablet_bootstrap.cc:492] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5: Bootstrap starting.
I20240503 08:11:09.637253 1432 raft_consensus.cc:348] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b770b3463d04b75a97b046626227b73" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 39909 } }
I20240503 08:11:09.637459 1432 raft_consensus.cc:374] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:09.637567 1432 raft_consensus.cc:729] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b770b3463d04b75a97b046626227b73, State: Initialized, Role: FOLLOWER
I20240503 08:11:09.637894 1432 consensus_queue.cc:260] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b770b3463d04b75a97b046626227b73" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 39909 } }
I20240503 08:11:09.638120 1432 raft_consensus.cc:388] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:09.638257 1432 raft_consensus.cc:482] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:09.638378 1432 raft_consensus.cc:3037] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:09.638484 1434 tablet_bootstrap.cc:654] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:09.640416 1434 tablet_bootstrap.cc:492] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5: No bootstrap required, opened a new log
I20240503 08:11:09.640595 1434 ts_tablet_manager.cc:1367] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:09.641182 1434 raft_consensus.cc:348] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584356bd4c4047b7b32f16de7d96d3a5" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32909 } }
I20240503 08:11:09.641321 1434 raft_consensus.cc:374] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:09.641449 1434 raft_consensus.cc:729] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584356bd4c4047b7b32f16de7d96d3a5, State: Initialized, Role: FOLLOWER
I20240503 08:11:09.641681 1434 consensus_queue.cc:260] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584356bd4c4047b7b32f16de7d96d3a5" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32909 } }
I20240503 08:11:09.641759 1432 raft_consensus.cc:504] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b770b3463d04b75a97b046626227b73" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 39909 } }
I20240503 08:11:09.641880 1434 raft_consensus.cc:388] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:09.641960 1434 raft_consensus.cc:482] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:09.642068 1434 raft_consensus.cc:3037] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:09.642050 1432 leader_election.cc:302] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b770b3463d04b75a97b046626227b73; no voters:
I20240503 08:11:09.642616 1432 leader_election.cc:288] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:09.642765 1440 raft_consensus.cc:2781] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:09.643257 1436 raft_consensus.cc:686] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [term 1 LEADER]: Becoming Leader. State: Replica: c779bc9db6244945af9789bd5ea66221, State: Running, Role: LEADER
I20240503 08:11:09.643646 1436 consensus_queue.cc:237] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c779bc9db6244945af9789bd5ea66221" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44413 } }
I20240503 08:11:09.643944 1431 ts_tablet_manager.cc:1398] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221: Time spent starting tablet: real 0.019s user 0.005s sys 0.006s
I20240503 08:11:09.644948 1434 raft_consensus.cc:504] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584356bd4c4047b7b32f16de7d96d3a5" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32909 } }
I20240503 08:11:09.645272 1434 leader_election.cc:302] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 584356bd4c4047b7b32f16de7d96d3a5; no voters:
I20240503 08:11:09.645694 1432 ts_tablet_manager.cc:1398] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20240503 08:11:09.650317 1440 raft_consensus.cc:686] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [term 1 LEADER]: Becoming Leader. State: Replica: 5b770b3463d04b75a97b046626227b73, State: Running, Role: LEADER
I20240503 08:11:09.650633 1434 leader_election.cc:288] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:09.650825 1444 raft_consensus.cc:2781] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:09.650849 1440 consensus_queue.cc:237] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b770b3463d04b75a97b046626227b73" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 39909 } }
I20240503 08:11:09.652086 1444 raft_consensus.cc:686] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [term 1 LEADER]: Becoming Leader. State: Replica: 584356bd4c4047b7b32f16de7d96d3a5, State: Running, Role: LEADER
I20240503 08:11:09.652371 1444 consensus_queue.cc:237] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584356bd4c4047b7b32f16de7d96d3a5" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32909 } }
I20240503 08:11:09.651587 1434 ts_tablet_manager.cc:1398] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20240503 08:11:09.656212 1434 tablet_bootstrap.cc:492] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5: Bootstrap starting.
I20240503 08:11:09.660317 949 catalog_manager.cc:5484] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b770b3463d04b75a97b046626227b73 (127.0.230.3). New cstate: current_term: 1 leader_uuid: "5b770b3463d04b75a97b046626227b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b770b3463d04b75a97b046626227b73" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 39909 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:09.668731 1434 tablet_bootstrap.cc:654] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:09.677436 949 catalog_manager.cc:5484] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 reported cstate change: term changed from 0 to 1, leader changed from <none> to c779bc9db6244945af9789bd5ea66221 (127.0.230.2). New cstate: current_term: 1 leader_uuid: "c779bc9db6244945af9789bd5ea66221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c779bc9db6244945af9789bd5ea66221" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44413 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:09.684548 948 catalog_manager.cc:5484] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 584356bd4c4047b7b32f16de7d96d3a5 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "584356bd4c4047b7b32f16de7d96d3a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584356bd4c4047b7b32f16de7d96d3a5" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32909 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:09.707927 1434 tablet_bootstrap.cc:492] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5: No bootstrap required, opened a new log
I20240503 08:11:09.708122 1434 ts_tablet_manager.cc:1367] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.010s
I20240503 08:11:09.708943 1434 raft_consensus.cc:348] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584356bd4c4047b7b32f16de7d96d3a5" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32909 } }
I20240503 08:11:09.709147 1434 raft_consensus.cc:374] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:09.709226 1434 raft_consensus.cc:729] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584356bd4c4047b7b32f16de7d96d3a5, State: Initialized, Role: FOLLOWER
I20240503 08:11:09.709436 1434 consensus_queue.cc:260] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584356bd4c4047b7b32f16de7d96d3a5" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32909 } }
I20240503 08:11:09.709592 1434 raft_consensus.cc:388] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:09.709682 1434 raft_consensus.cc:482] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:09.709766 1434 raft_consensus.cc:3037] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:09.711611 1434 raft_consensus.cc:504] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584356bd4c4047b7b32f16de7d96d3a5" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32909 } }
I20240503 08:11:09.711818 1434 leader_election.cc:302] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 584356bd4c4047b7b32f16de7d96d3a5; no voters:
I20240503 08:11:09.712008 1434 leader_election.cc:288] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:09.712082 1444 raft_consensus.cc:2781] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:09.712673 1444 raft_consensus.cc:686] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [term 1 LEADER]: Becoming Leader. State: Replica: 584356bd4c4047b7b32f16de7d96d3a5, State: Running, Role: LEADER
I20240503 08:11:09.712698 1434 ts_tablet_manager.cc:1398] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20240503 08:11:09.712918 1444 consensus_queue.cc:237] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584356bd4c4047b7b32f16de7d96d3a5" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32909 } }
I20240503 08:11:09.715093 950 catalog_manager.cc:5484] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 584356bd4c4047b7b32f16de7d96d3a5 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "584356bd4c4047b7b32f16de7d96d3a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584356bd4c4047b7b32f16de7d96d3a5" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32909 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:09.749150 920 tablet_server.cc:178] TabletServer@127.0.230.1:0 shutting down...
I20240503 08:11:09.758846 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:09.759284 920 tablet_replica.cc:330] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5: stopping tablet replica
I20240503 08:11:09.759721 920 raft_consensus.cc:2229] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:09.759979 920 raft_consensus.cc:2258] T 9151fb6f49a14115b0326ec35ab0e680 P 584356bd4c4047b7b32f16de7d96d3a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:09.761418 920 tablet_replica.cc:330] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5: stopping tablet replica
I20240503 08:11:09.761554 920 raft_consensus.cc:2229] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:09.761679 920 raft_consensus.cc:2258] T 03abec94ec6a4ecf8af1a987d115f2f1 P 584356bd4c4047b7b32f16de7d96d3a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:09.776588 920 tablet_server.cc:194] TabletServer@127.0.230.1:0 shutdown complete.
I20240503 08:11:09.781626 920 tablet_server.cc:178] TabletServer@127.0.230.2:0 shutting down...
I20240503 08:11:09.789149 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:09.789474 920 tablet_replica.cc:330] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221: stopping tablet replica
I20240503 08:11:09.789685 920 raft_consensus.cc:2229] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:09.789872 920 raft_consensus.cc:2258] T ace966a71f5e4bd0adb7e4dd209a6188 P c779bc9db6244945af9789bd5ea66221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:09.793730 920 tablet_server.cc:194] TabletServer@127.0.230.2:0 shutdown complete.
I20240503 08:11:09.796272 920 tablet_server.cc:178] TabletServer@127.0.230.3:0 shutting down...
I20240503 08:11:09.803598 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:09.803930 920 tablet_replica.cc:330] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73: stopping tablet replica
I20240503 08:11:09.804109 920 raft_consensus.cc:2229] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:09.804260 920 raft_consensus.cc:2258] T 799dcf6148774ecfb4cfc15207274bff P 5b770b3463d04b75a97b046626227b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:09.807909 920 tablet_server.cc:194] TabletServer@127.0.230.3:0 shutdown complete.
I20240503 08:11:09.810629 920 master.cc:537] Master@127.0.230.62:44059 shutting down...
I20240503 08:11:09.818289 920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:09.818686 920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:09.818826 920 tablet_replica.cc:330] T 00000000000000000000000000000000 P d701dd16e12b424cafeb6c93949c3499: stopping tablet replica
I20240503 08:11:09.833609 920 master.cc:552] Master@127.0.230.62:44059 shutdown complete.
I20240503 08:11:09.838404 920 master.cc:537] Master@127.0.230.61:42425 shutting down...
I20240503 08:11:09.845162 920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:09.845382 920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:09.845489 920 tablet_replica.cc:330] T 00000000000000000000000000000000 P cd67ee33153c457a87de4e87dec40947: stopping tablet replica
I20240503 08:11:09.859202 920 master.cc:552] Master@127.0.230.61:42425 shutdown complete.
I20240503 08:11:09.862938 920 master.cc:537] Master@127.0.230.60:45915 shutting down...
I20240503 08:11:09.869479 920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:09.869688 920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:09.869814 920 tablet_replica.cc:330] T 00000000000000000000000000000000 P 42501d4ad9334f36b3211d7b6f0622dd: stopping tablet replica
I20240503 08:11:09.883466 920 master.cc:552] Master@127.0.230.60:45915 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3102 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240503 08:11:09.895851 920 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.230.62:35749
I20240503 08:11:09.896407 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:09.898876 1457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:09.899371 1458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:09.900076 920 server_base.cc:1034] running on GCE node
W20240503 08:11:09.900517 1460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:09.900843 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:09.900918 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:09.900959 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723869900947 us; error 0 us; skew 500 ppm
I20240503 08:11:09.901116 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:09.902243 920 webserver.cc:460] Webserver started at http://127.0.230.62:36273/ using document root <none> and password file <none>
I20240503 08:11:09.902452 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:09.902523 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:09.902647 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:09.903239 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/master-0-root/instance:
uuid: "34991c958d30454fb341102af7fa8fcc"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-2jf9"
I20240503 08:11:09.905632 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:09.907155 1465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:09.907480 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:09.907601 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/master-0-root
uuid: "34991c958d30454fb341102af7fa8fcc"
format_stamp: "Formatted at 2024-05-03 08:11:09 on dist-test-slave-2jf9"
I20240503 08:11:09.907716 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:09.916857 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:09.917371 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:09.930160 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.62:35749
I20240503 08:11:09.930188 1516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.62:35749 every 8 connection(s)
I20240503 08:11:09.931465 1517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:09.935346 1517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc: Bootstrap starting.
I20240503 08:11:09.936558 1517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:09.938030 1517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc: No bootstrap required, opened a new log
I20240503 08:11:09.938596 1517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34991c958d30454fb341102af7fa8fcc" member_type: VOTER }
I20240503 08:11:09.938725 1517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:09.938820 1517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34991c958d30454fb341102af7fa8fcc, State: Initialized, Role: FOLLOWER
I20240503 08:11:09.939013 1517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34991c958d30454fb341102af7fa8fcc" member_type: VOTER }
I20240503 08:11:09.939159 1517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:09.939229 1517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:09.939334 1517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:09.941072 1517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34991c958d30454fb341102af7fa8fcc" member_type: VOTER }
I20240503 08:11:09.941304 1517 leader_election.cc:302] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34991c958d30454fb341102af7fa8fcc; no voters:
I20240503 08:11:09.941684 1517 leader_election.cc:288] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:09.941797 1520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:09.942245 1520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [term 1 LEADER]: Becoming Leader. State: Replica: 34991c958d30454fb341102af7fa8fcc, State: Running, Role: LEADER
I20240503 08:11:09.942507 1517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:09.942487 1520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34991c958d30454fb341102af7fa8fcc" member_type: VOTER }
I20240503 08:11:09.943351 1521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34991c958d30454fb341102af7fa8fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34991c958d30454fb341102af7fa8fcc" member_type: VOTER } }
I20240503 08:11:09.943435 1522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34991c958d30454fb341102af7fa8fcc. Latest consensus state: current_term: 1 leader_uuid: "34991c958d30454fb341102af7fa8fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34991c958d30454fb341102af7fa8fcc" member_type: VOTER } }
I20240503 08:11:09.943619 1521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:09.943648 1522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:09.944526 1525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:09.945762 1525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:09.949015 1525 catalog_manager.cc:1347] Generated new cluster ID: b6b74ddd430f42c8a114991fb4a6c72b
I20240503 08:11:09.949144 1525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:09.949533 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:11:09.977348 1525 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:09.978142 1525 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:09.986169 1525 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc: Generated new TSK 0
I20240503 08:11:09.986429 1525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:10.014348 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:10.017202 1538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:10.017776 1539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:10.018016 920 server_base.cc:1034] running on GCE node
W20240503 08:11:10.018316 1541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:10.018638 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:10.018702 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:10.018754 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723870018729 us; error 0 us; skew 500 ppm
I20240503 08:11:10.018872 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:10.020119 920 webserver.cc:460] Webserver started at http://127.0.230.1:46119/ using document root <none> and password file <none>
I20240503 08:11:10.020335 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:10.020411 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:10.020515 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:10.021163 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-0-root/instance:
uuid: "bba5e9184a07463a9c52576421281521"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-2jf9"
I20240503 08:11:10.023542 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:10.025046 1546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:10.025342 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:10.025435 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-0-root
uuid: "bba5e9184a07463a9c52576421281521"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-2jf9"
I20240503 08:11:10.025557 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:10.055117 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:10.055760 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:10.056317 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:10.057173 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:10.057238 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:10.057322 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:10.057365 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:10.071779 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.1:40443
I20240503 08:11:10.071807 1608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.1:40443 every 8 connection(s)
I20240503 08:11:10.073460 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:10.076351 1613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:10.076858 1614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:10.077334 1616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:10.078246 920 server_base.cc:1034] running on GCE node
I20240503 08:11:10.078516 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:10.078608 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:10.078677 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723870078652 us; error 0 us; skew 500 ppm
I20240503 08:11:10.078886 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:10.080756 1609 heartbeater.cc:347] Connected to a master server at 127.0.230.62:35749
I20240503 08:11:10.080889 1609 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:10.081269 1609 heartbeater.cc:511] Master 127.0.230.62:35749 requested a full tablet report, sending...
I20240503 08:11:10.082103 1482 ts_manager.cc:194] Registered new tserver with Master: bba5e9184a07463a9c52576421281521 (127.0.230.1:40443)
I20240503 08:11:10.083155 1482 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39438
I20240503 08:11:10.086344 920 webserver.cc:460] Webserver started at http://127.0.230.2:42437/ using document root <none> and password file <none>
I20240503 08:11:10.086577 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:10.086655 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:10.086791 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:10.087414 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-1-root/instance:
uuid: "af90f15247b04a15a349855ad8e603a6"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-2jf9"
I20240503 08:11:10.089787 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:10.091282 1621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:10.091632 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:10.091737 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-1-root
uuid: "af90f15247b04a15a349855ad8e603a6"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-2jf9"
I20240503 08:11:10.091846 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:10.105118 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:10.105700 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:10.106287 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:10.107142 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:10.107231 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:10.107322 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:10.107415 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:10.120693 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.2:35187
I20240503 08:11:10.120726 1683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.2:35187 every 8 connection(s)
I20240503 08:11:10.122337 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:10.126361 1687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:10.127122 1688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:10.128331 1690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:10.129429 1684 heartbeater.cc:347] Connected to a master server at 127.0.230.62:35749
I20240503 08:11:10.129621 1684 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:10.129624 920 server_base.cc:1034] running on GCE node
I20240503 08:11:10.129956 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:10.130035 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:10.130059 1684 heartbeater.cc:511] Master 127.0.230.62:35749 requested a full tablet report, sending...
I20240503 08:11:10.130110 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723870130073 us; error 0 us; skew 500 ppm
I20240503 08:11:10.130255 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:10.130939 1482 ts_manager.cc:194] Registered new tserver with Master: af90f15247b04a15a349855ad8e603a6 (127.0.230.2:35187)
I20240503 08:11:10.131609 920 webserver.cc:460] Webserver started at http://127.0.230.3:41931/ using document root <none> and password file <none>
I20240503 08:11:10.131784 1482 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39448
I20240503 08:11:10.131827 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:10.131927 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:10.132056 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:10.132709 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-2-root/instance:
uuid: "93516627678c47c6b7c9b28e78ae78cf"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-2jf9"
I20240503 08:11:10.134981 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 08:11:10.136544 1695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:10.136927 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:10.137022 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-2-root
uuid: "93516627678c47c6b7c9b28e78ae78cf"
format_stamp: "Formatted at 2024-05-03 08:11:10 on dist-test-slave-2jf9"
I20240503 08:11:10.137132 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:10.162686 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:10.163296 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:10.163902 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:10.164723 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:10.164796 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:10.164877 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:10.164918 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:10.178010 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.3:44869
I20240503 08:11:10.178040 1757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.3:44869 every 8 connection(s)
I20240503 08:11:10.184686 1758 heartbeater.cc:347] Connected to a master server at 127.0.230.62:35749
I20240503 08:11:10.184844 1758 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:10.185209 1758 heartbeater.cc:511] Master 127.0.230.62:35749 requested a full tablet report, sending...
I20240503 08:11:10.186082 1482 ts_manager.cc:194] Registered new tserver with Master: 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:10.186672 920 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00778533s
I20240503 08:11:10.186838 1482 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39462
I20240503 08:11:11.084421 1609 heartbeater.cc:503] Master 127.0.230.62:35749 was elected leader, sending a full tablet report...
I20240503 08:11:11.132982 1684 heartbeater.cc:503] Master 127.0.230.62:35749 was elected leader, sending a full tablet report...
I20240503 08:11:11.188267 1758 heartbeater.cc:503] Master 127.0.230.62:35749 was elected leader, sending a full tablet report...
I20240503 08:11:11.214461 920 test_util.cc:268] Using random seed: 1127959401
I20240503 08:11:11.225791 1482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39470:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:11:11.226540 1482 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:11:11.249583 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 8724626d99334661a58a321b6c178468 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 08:11:11.250030 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff28f2d0e2a4554ab9dd5240b87d83a (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 08:11:11.250222 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8724626d99334661a58a321b6c178468. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.250535 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff28f2d0e2a4554ab9dd5240b87d83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.250931 1647 tablet_service.cc:1467] Processing CreateTablet for tablet fefde46bada54868808a8c3eca2c730c (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 08:11:11.251358 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefde46bada54868808a8c3eca2c730c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.255853 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 41187e251c5f43e387e077c637891fe9 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 08:11:11.256224 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41187e251c5f43e387e077c637891fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.256165 1642 tablet_service.cc:1467] Processing CreateTablet for tablet c01a74d81a924520873129b0021e6253 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 08:11:11.256620 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c01a74d81a924520873129b0021e6253. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.257347 1644 tablet_service.cc:1467] Processing CreateTablet for tablet dab1033f8b31480c9bbdd9508656b0d0 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 08:11:11.249614 1648 tablet_service.cc:1467] Processing CreateTablet for tablet f7607f2c286d4ac29f373bd90f97b5d8 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 08:11:11.257759 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab1033f8b31480c9bbdd9508656b0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.258073 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7607f2c286d4ac29f373bd90f97b5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.258790 1643 tablet_service.cc:1467] Processing CreateTablet for tablet 847daf4866574864a071399c43a89f1c (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 08:11:11.259251 1643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847daf4866574864a071399c43a89f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.262128 1573 tablet_service.cc:1467] Processing CreateTablet for tablet f7607f2c286d4ac29f373bd90f97b5d8 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 08:11:11.262856 1573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7607f2c286d4ac29f373bd90f97b5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.264051 1778 tablet_bootstrap.cc:492] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6: Bootstrap starting.
I20240503 08:11:11.264242 1574 tablet_service.cc:1467] Processing CreateTablet for tablet 8724626d99334661a58a321b6c178468 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 08:11:11.264582 1574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8724626d99334661a58a321b6c178468. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.265415 1778 tablet_bootstrap.cc:654] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.266278 1571 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff28f2d0e2a4554ab9dd5240b87d83a (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 08:11:11.266258 1572 tablet_service.cc:1467] Processing CreateTablet for tablet fefde46bada54868808a8c3eca2c730c (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 08:11:11.266407 1721 tablet_service.cc:1467] Processing CreateTablet for tablet fefde46bada54868808a8c3eca2c730c (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 08:11:11.266801 1571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff28f2d0e2a4554ab9dd5240b87d83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.266944 1721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefde46bada54868808a8c3eca2c730c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.270066 1723 tablet_service.cc:1467] Processing CreateTablet for tablet 8724626d99334661a58a321b6c178468 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 08:11:11.270519 1723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8724626d99334661a58a321b6c178468. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.272831 1572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefde46bada54868808a8c3eca2c730c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.280197 1570 tablet_service.cc:1467] Processing CreateTablet for tablet 41187e251c5f43e387e077c637891fe9 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 08:11:11.281111 1722 tablet_service.cc:1467] Processing CreateTablet for tablet f7607f2c286d4ac29f373bd90f97b5d8 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 08:11:11.283309 1778 tablet_bootstrap.cc:492] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6: No bootstrap required, opened a new log
I20240503 08:11:11.283527 1778 ts_tablet_manager.cc:1367] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.008s
I20240503 08:11:11.283648 1722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7607f2c286d4ac29f373bd90f97b5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.283850 1717 tablet_service.cc:1467] Processing CreateTablet for tablet 847daf4866574864a071399c43a89f1c (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 08:11:11.284862 1567 tablet_service.cc:1467] Processing CreateTablet for tablet c01a74d81a924520873129b0021e6253 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 08:11:11.285902 1569 tablet_service.cc:1467] Processing CreateTablet for tablet dab1033f8b31480c9bbdd9508656b0d0 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 08:11:11.286458 1570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41187e251c5f43e387e077c637891fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.285754 1719 tablet_service.cc:1467] Processing CreateTablet for tablet 41187e251c5f43e387e077c637891fe9 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 08:11:11.286540 1568 tablet_service.cc:1467] Processing CreateTablet for tablet 847daf4866574864a071399c43a89f1c (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 08:11:11.288653 1720 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff28f2d0e2a4554ab9dd5240b87d83a (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 08:11:11.290267 1567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c01a74d81a924520873129b0021e6253. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.290441 1717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847daf4866574864a071399c43a89f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.289532 1718 tablet_service.cc:1467] Processing CreateTablet for tablet dab1033f8b31480c9bbdd9508656b0d0 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 08:11:11.291565 1568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847daf4866574864a071399c43a89f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.292914 1716 tablet_service.cc:1467] Processing CreateTablet for tablet c01a74d81a924520873129b0021e6253 (DEFAULT_TABLE table=test-workload [id=e9143301d1014fb3bc1a9104ab16cc45]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 08:11:11.293774 1719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41187e251c5f43e387e077c637891fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.294500 1569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab1033f8b31480c9bbdd9508656b0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.296389 1778 raft_consensus.cc:348] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } }
I20240503 08:11:11.298570 1778 raft_consensus.cc:374] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.298996 1720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff28f2d0e2a4554ab9dd5240b87d83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.301179 1718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab1033f8b31480c9bbdd9508656b0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.302800 1716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c01a74d81a924520873129b0021e6253. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.308307 1781 tablet_bootstrap.cc:492] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf: Bootstrap starting.
I20240503 08:11:11.305797 1778 raft_consensus.cc:729] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.310535 1778 consensus_queue.cc:260] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } }
I20240503 08:11:11.320523 1780 tablet_bootstrap.cc:492] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521: Bootstrap starting.
I20240503 08:11:11.323419 1778 ts_tablet_manager.cc:1398] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6: Time spent starting tablet: real 0.034s user 0.003s sys 0.001s
I20240503 08:11:11.324885 1778 tablet_bootstrap.cc:492] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6: Bootstrap starting.
I20240503 08:11:11.341876 1781 tablet_bootstrap.cc:654] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.341925 1778 tablet_bootstrap.cc:654] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.342339 1780 tablet_bootstrap.cc:654] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.344054 1778 tablet_bootstrap.cc:492] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6: No bootstrap required, opened a new log
I20240503 08:11:11.344256 1781 tablet_bootstrap.cc:492] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf: No bootstrap required, opened a new log
I20240503 08:11:11.344278 1778 ts_tablet_manager.cc:1367] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20240503 08:11:11.344426 1781 ts_tablet_manager.cc:1367] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240503 08:11:11.345351 1780 tablet_bootstrap.cc:492] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521: No bootstrap required, opened a new log
I20240503 08:11:11.345302 1781 raft_consensus.cc:348] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } }
I20240503 08:11:11.345281 1778 raft_consensus.cc:348] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.345490 1780 ts_tablet_manager.cc:1367] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.005s
I20240503 08:11:11.345535 1781 raft_consensus.cc:374] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.345566 1778 raft_consensus.cc:374] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.345649 1778 raft_consensus.cc:729] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.345654 1781 raft_consensus.cc:729] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93516627678c47c6b7c9b28e78ae78cf, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.345942 1781 consensus_queue.cc:260] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } }
I20240503 08:11:11.345942 1778 consensus_queue.cc:260] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.346247 1780 raft_consensus.cc:348] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.346365 1782 raft_consensus.cc:482] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:11.346550 1780 raft_consensus.cc:374] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.346541 1782 raft_consensus.cc:504] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.346735 1780 raft_consensus.cc:729] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba5e9184a07463a9c52576421281521, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.347069 1780 consensus_queue.cc:260] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.348624 1781 ts_tablet_manager.cc:1398] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20240503 08:11:11.348639 1778 ts_tablet_manager.cc:1398] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:11:11.349064 1781 tablet_bootstrap.cc:492] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf: Bootstrap starting.
I20240503 08:11:11.349244 1778 tablet_bootstrap.cc:492] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6: Bootstrap starting.
I20240503 08:11:11.350198 1782 leader_election.cc:288] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bba5e9184a07463a9c52576421281521 (127.0.230.1:40443), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.350687 1778 tablet_bootstrap.cc:654] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.351459 1780 ts_tablet_manager.cc:1398] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20240503 08:11:11.351886 1780 tablet_bootstrap.cc:492] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521: Bootstrap starting.
I20240503 08:11:11.359014 1781 tablet_bootstrap.cc:654] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.361898 1778 tablet_bootstrap.cc:492] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6: No bootstrap required, opened a new log
I20240503 08:11:11.362092 1778 ts_tablet_manager.cc:1367] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240503 08:11:11.362402 1780 tablet_bootstrap.cc:654] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.363029 1778 raft_consensus.cc:348] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.363309 1778 raft_consensus.cc:374] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.363469 1778 raft_consensus.cc:729] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.363806 1778 consensus_queue.cc:260] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.364387 1778 ts_tablet_manager.cc:1398] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 08:11:11.364665 1778 tablet_bootstrap.cc:492] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6: Bootstrap starting.
I20240503 08:11:11.366438 1778 tablet_bootstrap.cc:654] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.366644 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff28f2d0e2a4554ab9dd5240b87d83a" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba5e9184a07463a9c52576421281521" is_pre_election: true
I20240503 08:11:11.367038 1584 raft_consensus.cc:2454] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 0.
I20240503 08:11:11.368034 1622 leader_election.cc:302] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af90f15247b04a15a349855ad8e603a6, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:11.368330 1782 raft_consensus.cc:2781] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:11.368454 1782 raft_consensus.cc:482] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:11.368564 1782 raft_consensus.cc:3037] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.369664 1778 tablet_bootstrap.cc:492] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6: No bootstrap required, opened a new log
I20240503 08:11:11.369926 1778 ts_tablet_manager.cc:1367] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240503 08:11:11.370766 1778 raft_consensus.cc:348] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.371062 1778 raft_consensus.cc:374] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.371198 1778 raft_consensus.cc:729] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.371572 1778 consensus_queue.cc:260] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.371768 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff28f2d0e2a4554ab9dd5240b87d83a" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf" is_pre_election: true
I20240503 08:11:11.372315 1778 ts_tablet_manager.cc:1398] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 08:11:11.372697 1622 leader_election.cc:341] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:11.373071 1778 tablet_bootstrap.cc:492] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6: Bootstrap starting.
I20240503 08:11:11.374106 1780 tablet_bootstrap.cc:492] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521: No bootstrap required, opened a new log
I20240503 08:11:11.374281 1780 ts_tablet_manager.cc:1367] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.004s
I20240503 08:11:11.374676 1781 tablet_bootstrap.cc:492] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf: No bootstrap required, opened a new log
I20240503 08:11:11.374789 1778 tablet_bootstrap.cc:654] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.375589 1781 ts_tablet_manager.cc:1367] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.003s
I20240503 08:11:11.376081 1782 raft_consensus.cc:504] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.376380 1781 raft_consensus.cc:348] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.376590 1781 raft_consensus.cc:374] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.376616 1782 leader_election.cc:288] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Requested vote from peers bba5e9184a07463a9c52576421281521 (127.0.230.1:40443), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.376688 1781 raft_consensus.cc:729] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93516627678c47c6b7c9b28e78ae78cf, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.375237 1780 raft_consensus.cc:348] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } }
I20240503 08:11:11.376832 1780 raft_consensus.cc:374] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.376945 1780 raft_consensus.cc:729] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba5e9184a07463a9c52576421281521, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.377260 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff28f2d0e2a4554ab9dd5240b87d83a" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba5e9184a07463a9c52576421281521"
I20240503 08:11:11.377319 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff28f2d0e2a4554ab9dd5240b87d83a" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf"
I20240503 08:11:11.377435 1584 raft_consensus.cc:3037] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.378057 1778 tablet_bootstrap.cc:492] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6: No bootstrap required, opened a new log
I20240503 08:11:11.378172 1778 ts_tablet_manager.cc:1367] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:11:11.378746 1778 raft_consensus.cc:348] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.378898 1778 raft_consensus.cc:374] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.378979 1778 raft_consensus.cc:729] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.379141 1778 consensus_queue.cc:260] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.379596 1778 ts_tablet_manager.cc:1398] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
W20240503 08:11:11.379683 1622 leader_election.cc:341] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:11.379778 1584 raft_consensus.cc:2454] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 1.
I20240503 08:11:11.379894 1778 tablet_bootstrap.cc:492] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6: Bootstrap starting.
I20240503 08:11:11.379889 1781 consensus_queue.cc:260] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.380184 1622 leader_election.cc:302] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: af90f15247b04a15a349855ad8e603a6, bba5e9184a07463a9c52576421281521; no voters: 93516627678c47c6b7c9b28e78ae78cf
I20240503 08:11:11.380476 1781 ts_tablet_manager.cc:1398] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:11:11.380625 1782 raft_consensus.cc:2781] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:11.380810 1781 tablet_bootstrap.cc:492] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf: Bootstrap starting.
I20240503 08:11:11.380843 1782 raft_consensus.cc:686] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Becoming Leader. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Running, Role: LEADER
I20240503 08:11:11.381100 1782 consensus_queue.cc:237] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.382450 1781 tablet_bootstrap.cc:654] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.382596 1780 consensus_queue.cc:260] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } }
I20240503 08:11:11.383818 1780 ts_tablet_manager.cc:1398] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 08:11:11.384153 1780 tablet_bootstrap.cc:492] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521: Bootstrap starting.
I20240503 08:11:11.385628 1780 tablet_bootstrap.cc:654] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.386351 1778 tablet_bootstrap.cc:654] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.385948 1480 catalog_manager.cc:5484] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to af90f15247b04a15a349855ad8e603a6 (127.0.230.2). New cstate: current_term: 1 leader_uuid: "af90f15247b04a15a349855ad8e603a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:11.387068 1782 raft_consensus.cc:482] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:11.387305 1782 raft_consensus.cc:504] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } }
I20240503 08:11:11.388281 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8724626d99334661a58a321b6c178468" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf" is_pre_election: true
I20240503 08:11:11.388478 1733 raft_consensus.cc:2454] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 0.
I20240503 08:11:11.389078 1622 leader_election.cc:302] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93516627678c47c6b7c9b28e78ae78cf, af90f15247b04a15a349855ad8e603a6; no voters:
I20240503 08:11:11.389393 1787 raft_consensus.cc:2781] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:11.389484 1787 raft_consensus.cc:482] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:11.389583 1787 raft_consensus.cc:3037] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.391629 1781 tablet_bootstrap.cc:492] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf: No bootstrap required, opened a new log
I20240503 08:11:11.391773 1781 ts_tablet_manager.cc:1367] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240503 08:11:11.392513 1781 raft_consensus.cc:348] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.392660 1781 raft_consensus.cc:374] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.392725 1781 raft_consensus.cc:729] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93516627678c47c6b7c9b28e78ae78cf, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.392689 1787 raft_consensus.cc:504] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } }
I20240503 08:11:11.392947 1780 tablet_bootstrap.cc:492] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521: No bootstrap required, opened a new log
I20240503 08:11:11.393100 1780 ts_tablet_manager.cc:1367] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 08:11:11.392952 1781 consensus_queue.cc:260] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.393400 1787 leader_election.cc:288] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Requested vote from peers 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869), bba5e9184a07463a9c52576421281521 (127.0.230.1:40443)
I20240503 08:11:11.393678 1781 ts_tablet_manager.cc:1398] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:11.393958 1781 tablet_bootstrap.cc:492] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf: Bootstrap starting.
I20240503 08:11:11.393872 1780 raft_consensus.cc:348] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.394050 1780 raft_consensus.cc:374] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.394156 1780 raft_consensus.cc:729] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba5e9184a07463a9c52576421281521, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.394462 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8724626d99334661a58a321b6c178468" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba5e9184a07463a9c52576421281521"
I20240503 08:11:11.394440 1780 consensus_queue.cc:260] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.394685 1584 raft_consensus.cc:3037] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.395010 1780 ts_tablet_manager.cc:1398] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:11.395303 1780 tablet_bootstrap.cc:492] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521: Bootstrap starting.
I20240503 08:11:11.396157 1781 tablet_bootstrap.cc:654] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.396864 1780 tablet_bootstrap.cc:654] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.397568 1584 raft_consensus.cc:2454] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 1.
I20240503 08:11:11.397913 1778 tablet_bootstrap.cc:492] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6: No bootstrap required, opened a new log
I20240503 08:11:11.398083 1778 ts_tablet_manager.cc:1367] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240503 08:11:11.398680 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8724626d99334661a58a321b6c178468" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf"
I20240503 08:11:11.398805 1778 raft_consensus.cc:348] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.398913 1733 raft_consensus.cc:3037] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.399183 1622 leader_election.cc:302] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af90f15247b04a15a349855ad8e603a6, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:11.402007 1733 raft_consensus.cc:2454] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 1.
I20240503 08:11:11.402153 1787 raft_consensus.cc:2781] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:11.402585 1778 raft_consensus.cc:374] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.402693 1778 raft_consensus.cc:729] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.403014 1778 consensus_queue.cc:260] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.404664 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8724626d99334661a58a321b6c178468" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba5e9184a07463a9c52576421281521" is_pre_election: true
I20240503 08:11:11.404995 1584 raft_consensus.cc:2362] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 1. Re-sending same reply.
I20240503 08:11:11.405099 1778 ts_tablet_manager.cc:1398] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20240503 08:11:11.405660 1778 tablet_bootstrap.cc:492] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6: Bootstrap starting.
I20240503 08:11:11.405687 1782 leader_election.cc:288] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869), bba5e9184a07463a9c52576421281521 (127.0.230.1:40443)
I20240503 08:11:11.407343 1778 tablet_bootstrap.cc:654] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.408489 1787 raft_consensus.cc:686] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Becoming Leader. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Running, Role: LEADER
I20240503 08:11:11.408989 1787 consensus_queue.cc:237] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } }
I20240503 08:11:11.411819 1778 tablet_bootstrap.cc:492] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6: No bootstrap required, opened a new log
I20240503 08:11:11.411971 1778 ts_tablet_manager.cc:1367] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 08:11:11.412034 1480 catalog_manager.cc:5484] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to af90f15247b04a15a349855ad8e603a6 (127.0.230.2). New cstate: current_term: 1 leader_uuid: "af90f15247b04a15a349855ad8e603a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:11.412793 1778 raft_consensus.cc:348] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.412982 1778 raft_consensus.cc:374] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.413070 1778 raft_consensus.cc:729] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.413312 1778 consensus_queue.cc:260] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.413847 1778 ts_tablet_manager.cc:1398] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:11.414139 1778 tablet_bootstrap.cc:492] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6: Bootstrap starting.
I20240503 08:11:11.415988 1778 tablet_bootstrap.cc:654] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.417913 1781 tablet_bootstrap.cc:492] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf: No bootstrap required, opened a new log
I20240503 08:11:11.418084 1781 ts_tablet_manager.cc:1367] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.006s
I20240503 08:11:11.418412 1780 tablet_bootstrap.cc:492] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521: No bootstrap required, opened a new log
I20240503 08:11:11.418578 1780 ts_tablet_manager.cc:1367] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.011s
I20240503 08:11:11.418738 1778 tablet_bootstrap.cc:492] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6: No bootstrap required, opened a new log
I20240503 08:11:11.418800 1781 raft_consensus.cc:348] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.418900 1778 ts_tablet_manager.cc:1367] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 08:11:11.418992 1781 raft_consensus.cc:374] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.419078 1781 raft_consensus.cc:729] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93516627678c47c6b7c9b28e78ae78cf, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.419277 1780 raft_consensus.cc:348] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.419385 1781 consensus_queue.cc:260] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.419564 1780 raft_consensus.cc:374] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.419653 1780 raft_consensus.cc:729] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba5e9184a07463a9c52576421281521, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.419750 1778 raft_consensus.cc:348] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.419987 1778 raft_consensus.cc:374] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.419939 1780 consensus_queue.cc:260] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.420075 1778 raft_consensus.cc:729] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.420459 1780 ts_tablet_manager.cc:1398] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:11.420531 1778 consensus_queue.cc:260] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.420753 1780 tablet_bootstrap.cc:492] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521: Bootstrap starting.
I20240503 08:11:11.421945 1781 ts_tablet_manager.cc:1398] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:11:11.422387 1780 tablet_bootstrap.cc:654] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.423046 1778 ts_tablet_manager.cc:1398] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:11:11.423946 1781 tablet_bootstrap.cc:492] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf: Bootstrap starting.
I20240503 08:11:11.425529 1781 tablet_bootstrap.cc:654] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.427162 1781 tablet_bootstrap.cc:492] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf: No bootstrap required, opened a new log
I20240503 08:11:11.427287 1781 ts_tablet_manager.cc:1367] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:11.427950 1781 raft_consensus.cc:348] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.428121 1781 raft_consensus.cc:374] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.428217 1781 raft_consensus.cc:729] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93516627678c47c6b7c9b28e78ae78cf, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.428328 1780 tablet_bootstrap.cc:492] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521: No bootstrap required, opened a new log
I20240503 08:11:11.428462 1780 ts_tablet_manager.cc:1367] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20240503 08:11:11.428424 1781 consensus_queue.cc:260] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.428881 1781 ts_tablet_manager.cc:1398] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:11.429145 1781 tablet_bootstrap.cc:492] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf: Bootstrap starting.
I20240503 08:11:11.429098 1780 raft_consensus.cc:348] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.429234 1780 raft_consensus.cc:374] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.429296 1780 raft_consensus.cc:729] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba5e9184a07463a9c52576421281521, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.429471 1780 consensus_queue.cc:260] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.430059 1780 ts_tablet_manager.cc:1398] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 08:11:11.430315 1780 tablet_bootstrap.cc:492] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521: Bootstrap starting.
I20240503 08:11:11.430694 1781 tablet_bootstrap.cc:654] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.431936 1780 tablet_bootstrap.cc:654] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.433848 1780 tablet_bootstrap.cc:492] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521: No bootstrap required, opened a new log
I20240503 08:11:11.434007 1780 ts_tablet_manager.cc:1367] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:11.434703 1780 raft_consensus.cc:348] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.434862 1780 raft_consensus.cc:374] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.434926 1780 raft_consensus.cc:729] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba5e9184a07463a9c52576421281521, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.435060 1781 tablet_bootstrap.cc:492] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf: No bootstrap required, opened a new log
I20240503 08:11:11.435212 1781 ts_tablet_manager.cc:1367] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:11:11.435142 1780 consensus_queue.cc:260] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.435606 1780 ts_tablet_manager.cc:1398] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:11:11.435878 1780 tablet_bootstrap.cc:492] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521: Bootstrap starting.
I20240503 08:11:11.435906 1781 raft_consensus.cc:348] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.436118 1781 raft_consensus.cc:374] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.436201 1781 raft_consensus.cc:729] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93516627678c47c6b7c9b28e78ae78cf, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.436451 1781 consensus_queue.cc:260] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.436852 1781 ts_tablet_manager.cc:1398] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:11:11.437140 1781 tablet_bootstrap.cc:492] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf: Bootstrap starting.
I20240503 08:11:11.437327 1780 tablet_bootstrap.cc:654] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.438596 1781 tablet_bootstrap.cc:654] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.441335 1780 tablet_bootstrap.cc:492] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521: No bootstrap required, opened a new log
I20240503 08:11:11.441442 1780 ts_tablet_manager.cc:1367] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 08:11:11.441972 1780 raft_consensus.cc:348] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.442106 1780 raft_consensus.cc:374] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.442173 1780 raft_consensus.cc:729] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba5e9184a07463a9c52576421281521, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.442342 1780 consensus_queue.cc:260] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.442775 1780 ts_tablet_manager.cc:1398] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:11:11.442986 1780 tablet_bootstrap.cc:492] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521: Bootstrap starting.
I20240503 08:11:11.444185 1780 tablet_bootstrap.cc:654] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.444798 1788 raft_consensus.cc:482] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:11.444933 1788 raft_consensus.cc:504] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.446130 1788 leader_election.cc:288] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af90f15247b04a15a349855ad8e603a6 (127.0.230.2:35187), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.446477 1781 tablet_bootstrap.cc:492] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf: No bootstrap required, opened a new log
I20240503 08:11:11.446622 1781 ts_tablet_manager.cc:1367] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20240503 08:11:11.447448 1781 raft_consensus.cc:348] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.447659 1781 raft_consensus.cc:374] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.447793 1781 raft_consensus.cc:729] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93516627678c47c6b7c9b28e78ae78cf, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.448033 1781 consensus_queue.cc:260] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.450589 1781 ts_tablet_manager.cc:1398] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:11:11.452868 1781 tablet_bootstrap.cc:492] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf: Bootstrap starting.
I20240503 08:11:11.454496 1781 tablet_bootstrap.cc:654] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:11.460801 1780 tablet_bootstrap.cc:492] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521: No bootstrap required, opened a new log
I20240503 08:11:11.461091 1780 ts_tablet_manager.cc:1367] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20240503 08:11:11.461864 1780 raft_consensus.cc:348] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.462040 1780 raft_consensus.cc:374] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.462092 1780 raft_consensus.cc:729] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba5e9184a07463a9c52576421281521, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.462296 1780 consensus_queue.cc:260] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.462790 1780 ts_tablet_manager.cc:1398] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:11.465284 1781 tablet_bootstrap.cc:492] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf: No bootstrap required, opened a new log
I20240503 08:11:11.465482 1781 ts_tablet_manager.cc:1367] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 08:11:11.466336 1781 raft_consensus.cc:348] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.466560 1781 raft_consensus.cc:374] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:11.466650 1781 raft_consensus.cc:729] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93516627678c47c6b7c9b28e78ae78cf, State: Initialized, Role: FOLLOWER
I20240503 08:11:11.466876 1781 consensus_queue.cc:260] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.467830 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7607f2c286d4ac29f373bd90f97b5d8" candidate_uuid: "bba5e9184a07463a9c52576421281521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af90f15247b04a15a349855ad8e603a6" is_pre_election: true
I20240503 08:11:11.468063 1781 ts_tablet_manager.cc:1398] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:11:11.468163 1659 raft_consensus.cc:2454] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba5e9184a07463a9c52576421281521 in term 0.
I20240503 08:11:11.468591 1548 leader_election.cc:302] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af90f15247b04a15a349855ad8e603a6, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:11.468842 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7607f2c286d4ac29f373bd90f97b5d8" candidate_uuid: "bba5e9184a07463a9c52576421281521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf" is_pre_election: true
I20240503 08:11:11.469038 1733 raft_consensus.cc:2454] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba5e9184a07463a9c52576421281521 in term 0.
I20240503 08:11:11.469267 1788 raft_consensus.cc:2781] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:11.469389 1788 raft_consensus.cc:482] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:11.469480 1788 raft_consensus.cc:3037] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.471451 1788 raft_consensus.cc:504] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.472026 1788 leader_election.cc:288] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [CANDIDATE]: Term 1 election: Requested vote from peers af90f15247b04a15a349855ad8e603a6 (127.0.230.2:35187), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.472280 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7607f2c286d4ac29f373bd90f97b5d8" candidate_uuid: "bba5e9184a07463a9c52576421281521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af90f15247b04a15a349855ad8e603a6"
I20240503 08:11:11.472424 1659 raft_consensus.cc:3037] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.472378 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7607f2c286d4ac29f373bd90f97b5d8" candidate_uuid: "bba5e9184a07463a9c52576421281521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf"
I20240503 08:11:11.472577 1733 raft_consensus.cc:3037] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.474418 1659 raft_consensus.cc:2454] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba5e9184a07463a9c52576421281521 in term 1.
I20240503 08:11:11.474491 1733 raft_consensus.cc:2454] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba5e9184a07463a9c52576421281521 in term 1.
I20240503 08:11:11.474762 1548 leader_election.cc:302] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af90f15247b04a15a349855ad8e603a6, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:11.475044 1788 raft_consensus.cc:2781] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:11.475426 1788 raft_consensus.cc:686] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 1 LEADER]: Becoming Leader. State: Replica: bba5e9184a07463a9c52576421281521, State: Running, Role: LEADER
I20240503 08:11:11.475682 1788 consensus_queue.cc:237] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.477936 1479 catalog_manager.cc:5484] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 reported cstate change: term changed from 0 to 1, leader changed from <none> to bba5e9184a07463a9c52576421281521 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "bba5e9184a07463a9c52576421281521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:11.500793 1787 raft_consensus.cc:482] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:11.500967 1787 raft_consensus.cc:504] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.501569 1787 leader_election.cc:288] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bba5e9184a07463a9c52576421281521 (127.0.230.1:40443), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.502139 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab1033f8b31480c9bbdd9508656b0d0" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba5e9184a07463a9c52576421281521" is_pre_election: true
I20240503 08:11:11.502099 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab1033f8b31480c9bbdd9508656b0d0" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf" is_pre_election: true
I20240503 08:11:11.502341 1733 raft_consensus.cc:2454] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 0.
I20240503 08:11:11.502341 1584 raft_consensus.cc:2454] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 0.
I20240503 08:11:11.502755 1622 leader_election.cc:302] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93516627678c47c6b7c9b28e78ae78cf, af90f15247b04a15a349855ad8e603a6; no voters:
I20240503 08:11:11.503005 1787 raft_consensus.cc:2781] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:11.503122 1787 raft_consensus.cc:482] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:11.503217 1787 raft_consensus.cc:3037] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.505110 1787 raft_consensus.cc:504] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.505609 1787 leader_election.cc:288] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Requested vote from peers bba5e9184a07463a9c52576421281521 (127.0.230.1:40443), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.505990 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab1033f8b31480c9bbdd9508656b0d0" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba5e9184a07463a9c52576421281521"
I20240503 08:11:11.506048 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab1033f8b31480c9bbdd9508656b0d0" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf"
I20240503 08:11:11.506155 1584 raft_consensus.cc:3037] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.506199 1733 raft_consensus.cc:3037] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.508157 1733 raft_consensus.cc:2454] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 1.
I20240503 08:11:11.508211 1584 raft_consensus.cc:2454] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 1.
I20240503 08:11:11.508502 1622 leader_election.cc:302] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93516627678c47c6b7c9b28e78ae78cf, af90f15247b04a15a349855ad8e603a6; no voters:
I20240503 08:11:11.508765 1787 raft_consensus.cc:2781] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:11.508894 1787 raft_consensus.cc:686] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Becoming Leader. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Running, Role: LEADER
I20240503 08:11:11.509136 1787 consensus_queue.cc:237] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.511488 1479 catalog_manager.cc:5484] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to af90f15247b04a15a349855ad8e603a6 (127.0.230.2). New cstate: current_term: 1 leader_uuid: "af90f15247b04a15a349855ad8e603a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:11.517904 1787 raft_consensus.cc:482] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:11.518092 1787 raft_consensus.cc:504] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.518623 1787 leader_election.cc:288] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bba5e9184a07463a9c52576421281521 (127.0.230.1:40443), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.519066 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847daf4866574864a071399c43a89f1c" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba5e9184a07463a9c52576421281521" is_pre_election: true
I20240503 08:11:11.519105 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847daf4866574864a071399c43a89f1c" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf" is_pre_election: true
I20240503 08:11:11.519289 1584 raft_consensus.cc:2454] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 0.
I20240503 08:11:11.519311 1733 raft_consensus.cc:2454] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 0.
I20240503 08:11:11.519654 1622 leader_election.cc:302] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93516627678c47c6b7c9b28e78ae78cf, af90f15247b04a15a349855ad8e603a6; no voters:
I20240503 08:11:11.519905 1787 raft_consensus.cc:2781] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:11.519989 1787 raft_consensus.cc:482] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:11.520087 1787 raft_consensus.cc:3037] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.522248 1787 raft_consensus.cc:504] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.522754 1787 leader_election.cc:288] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Requested vote from peers bba5e9184a07463a9c52576421281521 (127.0.230.1:40443), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.523106 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847daf4866574864a071399c43a89f1c" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba5e9184a07463a9c52576421281521"
I20240503 08:11:11.523123 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847daf4866574864a071399c43a89f1c" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf"
I20240503 08:11:11.523296 1584 raft_consensus.cc:3037] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.523319 1733 raft_consensus.cc:3037] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.525287 1733 raft_consensus.cc:2454] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 1.
I20240503 08:11:11.525323 1584 raft_consensus.cc:2454] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 1.
I20240503 08:11:11.525658 1622 leader_election.cc:302] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af90f15247b04a15a349855ad8e603a6, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:11.525897 1787 raft_consensus.cc:2781] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:11.526041 1787 raft_consensus.cc:686] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Becoming Leader. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Running, Role: LEADER
I20240503 08:11:11.526288 1787 consensus_queue.cc:237] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.528676 1479 catalog_manager.cc:5484] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to af90f15247b04a15a349855ad8e603a6 (127.0.230.2). New cstate: current_term: 1 leader_uuid: "af90f15247b04a15a349855ad8e603a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:11.620990 1788 raft_consensus.cc:482] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:11.621209 1788 raft_consensus.cc:504] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.621840 1788 leader_election.cc:288] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af90f15247b04a15a349855ad8e603a6 (127.0.230.2:35187), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.622328 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefde46bada54868808a8c3eca2c730c" candidate_uuid: "bba5e9184a07463a9c52576421281521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af90f15247b04a15a349855ad8e603a6" is_pre_election: true
I20240503 08:11:11.622347 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefde46bada54868808a8c3eca2c730c" candidate_uuid: "bba5e9184a07463a9c52576421281521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf" is_pre_election: true
I20240503 08:11:11.622568 1733 raft_consensus.cc:2454] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba5e9184a07463a9c52576421281521 in term 0.
I20240503 08:11:11.622572 1659 raft_consensus.cc:2454] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba5e9184a07463a9c52576421281521 in term 0.
I20240503 08:11:11.623004 1547 leader_election.cc:302] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 93516627678c47c6b7c9b28e78ae78cf, af90f15247b04a15a349855ad8e603a6, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:11.623332 1788 raft_consensus.cc:2781] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:11.623474 1788 raft_consensus.cc:482] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:11.623562 1788 raft_consensus.cc:3037] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.625711 1788 raft_consensus.cc:504] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.626240 1788 leader_election.cc:288] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [CANDIDATE]: Term 1 election: Requested vote from peers af90f15247b04a15a349855ad8e603a6 (127.0.230.2:35187), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.626566 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefde46bada54868808a8c3eca2c730c" candidate_uuid: "bba5e9184a07463a9c52576421281521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af90f15247b04a15a349855ad8e603a6"
I20240503 08:11:11.626667 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefde46bada54868808a8c3eca2c730c" candidate_uuid: "bba5e9184a07463a9c52576421281521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf"
I20240503 08:11:11.626744 1659 raft_consensus.cc:3037] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.626843 1733 raft_consensus.cc:3037] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.628880 1733 raft_consensus.cc:2454] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba5e9184a07463a9c52576421281521 in term 1.
I20240503 08:11:11.628877 1659 raft_consensus.cc:2454] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba5e9184a07463a9c52576421281521 in term 1.
I20240503 08:11:11.629262 1547 leader_election.cc:302] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 93516627678c47c6b7c9b28e78ae78cf, af90f15247b04a15a349855ad8e603a6, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:11.629532 1788 raft_consensus.cc:2781] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:11.629671 1788 raft_consensus.cc:686] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 1 LEADER]: Becoming Leader. State: Replica: bba5e9184a07463a9c52576421281521, State: Running, Role: LEADER
I20240503 08:11:11.629892 1788 consensus_queue.cc:237] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.632314 1479 catalog_manager.cc:5484] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 reported cstate change: term changed from 0 to 1, leader changed from <none> to bba5e9184a07463a9c52576421281521 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "bba5e9184a07463a9c52576421281521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:11.665830 1787 raft_consensus.cc:482] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:11.666042 1787 raft_consensus.cc:504] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.666703 1787 leader_election.cc:288] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bba5e9184a07463a9c52576421281521 (127.0.230.1:40443), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.667266 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41187e251c5f43e387e077c637891fe9" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba5e9184a07463a9c52576421281521" is_pre_election: true
I20240503 08:11:11.667277 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41187e251c5f43e387e077c637891fe9" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf" is_pre_election: true
I20240503 08:11:11.667493 1584 raft_consensus.cc:2454] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 0.
I20240503 08:11:11.667549 1733 raft_consensus.cc:2454] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 0.
I20240503 08:11:11.667897 1622 leader_election.cc:302] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af90f15247b04a15a349855ad8e603a6, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:11.668159 1787 raft_consensus.cc:2781] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:11.668269 1787 raft_consensus.cc:482] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:11.668356 1787 raft_consensus.cc:3037] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.670243 1787 raft_consensus.cc:504] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.670795 1787 leader_election.cc:288] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Requested vote from peers bba5e9184a07463a9c52576421281521 (127.0.230.1:40443), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.671200 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41187e251c5f43e387e077c637891fe9" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba5e9184a07463a9c52576421281521"
I20240503 08:11:11.671221 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41187e251c5f43e387e077c637891fe9" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf"
I20240503 08:11:11.671365 1584 raft_consensus.cc:3037] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.671434 1733 raft_consensus.cc:3037] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.673358 1584 raft_consensus.cc:2454] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 1.
I20240503 08:11:11.673355 1733 raft_consensus.cc:2454] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 1.
I20240503 08:11:11.673709 1622 leader_election.cc:302] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93516627678c47c6b7c9b28e78ae78cf, af90f15247b04a15a349855ad8e603a6; no voters:
I20240503 08:11:11.673997 1787 raft_consensus.cc:2781] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:11.674127 1787 raft_consensus.cc:686] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Becoming Leader. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Running, Role: LEADER
I20240503 08:11:11.674368 1787 consensus_queue.cc:237] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.676870 1479 catalog_manager.cc:5484] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to af90f15247b04a15a349855ad8e603a6 (127.0.230.2). New cstate: current_term: 1 leader_uuid: "af90f15247b04a15a349855ad8e603a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:11.781903 1787 raft_consensus.cc:482] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:11.782119 1787 raft_consensus.cc:504] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.782806 1787 leader_election.cc:288] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bba5e9184a07463a9c52576421281521 (127.0.230.1:40443), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.783365 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01a74d81a924520873129b0021e6253" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba5e9184a07463a9c52576421281521" is_pre_election: true
I20240503 08:11:11.783432 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01a74d81a924520873129b0021e6253" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf" is_pre_election: true
I20240503 08:11:11.783651 1584 raft_consensus.cc:2454] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 0.
I20240503 08:11:11.783656 1733 raft_consensus.cc:2454] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 0.
I20240503 08:11:11.784034 1622 leader_election.cc:302] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af90f15247b04a15a349855ad8e603a6, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:11.784291 1787 raft_consensus.cc:2781] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:11.784368 1787 raft_consensus.cc:482] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:11.784461 1787 raft_consensus.cc:3037] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.786259 1787 raft_consensus.cc:504] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.786780 1787 leader_election.cc:288] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Requested vote from peers bba5e9184a07463a9c52576421281521 (127.0.230.1:40443), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:11.787180 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01a74d81a924520873129b0021e6253" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba5e9184a07463a9c52576421281521"
I20240503 08:11:11.787220 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01a74d81a924520873129b0021e6253" candidate_uuid: "af90f15247b04a15a349855ad8e603a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93516627678c47c6b7c9b28e78ae78cf"
I20240503 08:11:11.787364 1733 raft_consensus.cc:3037] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.787370 1584 raft_consensus.cc:3037] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.789271 1733 raft_consensus.cc:2454] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 1.
I20240503 08:11:11.789268 1584 raft_consensus.cc:2454] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af90f15247b04a15a349855ad8e603a6 in term 1.
I20240503 08:11:11.789633 1622 leader_election.cc:302] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af90f15247b04a15a349855ad8e603a6, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:11.789887 1787 raft_consensus.cc:2781] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:11.790046 1787 raft_consensus.cc:686] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Becoming Leader. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Running, Role: LEADER
I20240503 08:11:11.790262 1787 consensus_queue.cc:237] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:11.792831 1479 catalog_manager.cc:5484] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to af90f15247b04a15a349855ad8e603a6 (127.0.230.2). New cstate: current_term: 1 leader_uuid: "af90f15247b04a15a349855ad8e603a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:11.801611 1787 consensus_queue.cc:1035] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:11.805831 1787 consensus_queue.cc:1035] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:11.823581 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:11.826695 1829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:11.827253 1830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:11.827570 920 server_base.cc:1034] running on GCE node
W20240503 08:11:11.828199 1832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:11.828521 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:11.828608 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:11.828661 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723871828645 us; error 0 us; skew 500 ppm
I20240503 08:11:11.828832 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:11.834754 920 webserver.cc:460] Webserver started at http://127.0.230.4:33235/ using document root <none> and password file <none>
I20240503 08:11:11.835011 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:11.835088 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:11.835239 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:11.835942 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-3-root/instance:
uuid: "f1e919f7c4cb410dbcfb5970d30066bb"
format_stamp: "Formatted at 2024-05-03 08:11:11 on dist-test-slave-2jf9"
I20240503 08:11:11.838553 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 08:11:11.840212 1837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:11.840541 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:11.840651 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-3-root
uuid: "f1e919f7c4cb410dbcfb5970d30066bb"
format_stamp: "Formatted at 2024-05-03 08:11:11 on dist-test-slave-2jf9"
I20240503 08:11:11.840783 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723866726930-920-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:11.870898 1788 consensus_queue.cc:1035] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:11.871754 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:11.872514 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:11.873173 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:11.875213 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:11.875331 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:11.875495 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:11.875583 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:11.878340 1788 consensus_queue.cc:1035] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:11.882077 1732 raft_consensus.cc:3037] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:11.884789 1787 consensus_queue.cc:1035] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:11.889514 1803 consensus_queue.cc:1035] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:11.914038 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.4:39179
I20240503 08:11:11.914074 1904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.4:39179 every 8 connection(s)
I20240503 08:11:11.917047 1803 consensus_queue.cc:1035] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:11.917569 1787 consensus_queue.cc:1035] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:11.930212 1905 heartbeater.cc:347] Connected to a master server at 127.0.230.62:35749
I20240503 08:11:11.930370 1905 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:11.931506 1905 heartbeater.cc:511] Master 127.0.230.62:35749 requested a full tablet report, sending...
I20240503 08:11:11.932382 1479 ts_manager.cc:194] Registered new tserver with Master: f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4:39179)
I20240503 08:11:11.933426 1479 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39480
I20240503 08:11:11.949563 1536 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:11.957887 1659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "41187e251c5f43e387e077c637891fe9"
dest_uuid: "af90f15247b04a15a349855ad8e603a6"
mode: GRACEFUL
new_leader_uuid: "93516627678c47c6b7c9b28e78ae78cf"
from {username='slave'} at 127.0.0.1:32972
I20240503 08:11:11.958151 1659 raft_consensus.cc:595] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Received request to transfer leadership to TS 93516627678c47c6b7c9b28e78ae78cf
I20240503 08:11:11.959161 1659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "847daf4866574864a071399c43a89f1c"
dest_uuid: "af90f15247b04a15a349855ad8e603a6"
mode: GRACEFUL
new_leader_uuid: "bba5e9184a07463a9c52576421281521"
from {username='slave'} at 127.0.0.1:32972
I20240503 08:11:11.959323 1659 raft_consensus.cc:595] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Received request to transfer leadership to TS bba5e9184a07463a9c52576421281521
I20240503 08:11:11.960248 1659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8724626d99334661a58a321b6c178468"
dest_uuid: "af90f15247b04a15a349855ad8e603a6"
mode: GRACEFUL
new_leader_uuid: "93516627678c47c6b7c9b28e78ae78cf"
from {username='slave'} at 127.0.0.1:32972
I20240503 08:11:11.960419 1659 raft_consensus.cc:595] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Received request to transfer leadership to TS 93516627678c47c6b7c9b28e78ae78cf
I20240503 08:11:11.960816 1536 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 08:11:11.961190 1536 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:11.994396 1803 consensus_queue.cc:1035] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:11.996974 1782 raft_consensus.cc:979] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6: : Instructing follower bba5e9184a07463a9c52576421281521 to start an election
I20240503 08:11:11.997207 1803 raft_consensus.cc:1067] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Signalling peer bba5e9184a07463a9c52576421281521 to start an election
I20240503 08:11:11.998337 1583 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "847daf4866574864a071399c43a89f1c"
dest_uuid: "bba5e9184a07463a9c52576421281521"
from {username='slave'} at 127.0.0.1:50860
I20240503 08:11:11.999260 1782 consensus_queue.cc:1035] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:11.999361 1583 raft_consensus.cc:482] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:11.999665 1583 raft_consensus.cc:3037] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:12.002373 1583 raft_consensus.cc:504] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:12.003141 1583 leader_election.cc:288] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [CANDIDATE]: Term 2 election: Requested vote from peers af90f15247b04a15a349855ad8e603a6 (127.0.230.2:35187), 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3:44869)
I20240503 08:11:12.003499 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847daf4866574864a071399c43a89f1c" candidate_uuid: "bba5e9184a07463a9c52576421281521" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af90f15247b04a15a349855ad8e603a6"
I20240503 08:11:12.003536 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847daf4866574864a071399c43a89f1c" candidate_uuid: "bba5e9184a07463a9c52576421281521" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93516627678c47c6b7c9b28e78ae78cf"
I20240503 08:11:12.003720 1659 raft_consensus.cc:3032] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:12.003718 1733 raft_consensus.cc:3037] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:12.003857 1659 raft_consensus.cc:729] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Running, Role: LEADER
I20240503 08:11:12.004177 1659 consensus_queue.cc:260] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:12.004591 1659 raft_consensus.cc:3037] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:12.006608 1733 raft_consensus.cc:2454] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba5e9184a07463a9c52576421281521 in term 2.
I20240503 08:11:12.006974 1547 leader_election.cc:302] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93516627678c47c6b7c9b28e78ae78cf, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:12.007153 1659 raft_consensus.cc:2454] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba5e9184a07463a9c52576421281521 in term 2.
I20240503 08:11:12.007234 1846 raft_consensus.cc:2781] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:12.007367 1846 raft_consensus.cc:686] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 2 LEADER]: Becoming Leader. State: Replica: bba5e9184a07463a9c52576421281521, State: Running, Role: LEADER
I20240503 08:11:12.007696 1846 consensus_queue.cc:237] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:12.009647 1480 catalog_manager.cc:5484] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 reported cstate change: term changed from 1 to 2, leader changed from af90f15247b04a15a349855ad8e603a6 (127.0.230.2) to bba5e9184a07463a9c52576421281521 (127.0.230.1). New cstate: current_term: 2 leader_uuid: "bba5e9184a07463a9c52576421281521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.065905 1782 consensus_queue.cc:1035] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.069897 1787 consensus_queue.cc:1035] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.073251 1803 raft_consensus.cc:979] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6: : Instructing follower 93516627678c47c6b7c9b28e78ae78cf to start an election
I20240503 08:11:12.073443 1803 raft_consensus.cc:1067] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Signalling peer 93516627678c47c6b7c9b28e78ae78cf to start an election
I20240503 08:11:12.074047 1733 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "41187e251c5f43e387e077c637891fe9"
dest_uuid: "93516627678c47c6b7c9b28e78ae78cf"
from {username='slave'} at 127.0.0.1:60228
I20240503 08:11:12.074208 1733 raft_consensus.cc:482] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:12.074326 1733 raft_consensus.cc:3037] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:12.076259 1733 raft_consensus.cc:504] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:12.082866 1733 leader_election.cc:288] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [CANDIDATE]: Term 2 election: Requested vote from peers bba5e9184a07463a9c52576421281521 (127.0.230.1:40443), af90f15247b04a15a349855ad8e603a6 (127.0.230.2:35187)
I20240503 08:11:12.088743 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41187e251c5f43e387e077c637891fe9" candidate_uuid: "93516627678c47c6b7c9b28e78ae78cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bba5e9184a07463a9c52576421281521"
I20240503 08:11:12.088744 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41187e251c5f43e387e077c637891fe9" candidate_uuid: "93516627678c47c6b7c9b28e78ae78cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af90f15247b04a15a349855ad8e603a6"
I20240503 08:11:12.088966 1583 raft_consensus.cc:3037] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:12.089082 1659 raft_consensus.cc:3032] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:12.089196 1659 raft_consensus.cc:729] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Running, Role: LEADER
I20240503 08:11:12.089466 1659 consensus_queue.cc:260] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:12.089818 1659 raft_consensus.cc:3037] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:12.091099 1583 raft_consensus.cc:2454] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93516627678c47c6b7c9b28e78ae78cf in term 2.
I20240503 08:11:12.091497 1696 leader_election.cc:302] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93516627678c47c6b7c9b28e78ae78cf, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:12.091764 1786 raft_consensus.cc:2781] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:12.091907 1659 raft_consensus.cc:2454] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93516627678c47c6b7c9b28e78ae78cf in term 2.
I20240503 08:11:12.092388 1786 raft_consensus.cc:686] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 2 LEADER]: Becoming Leader. State: Replica: 93516627678c47c6b7c9b28e78ae78cf, State: Running, Role: LEADER
I20240503 08:11:12.092756 1786 consensus_queue.cc:237] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } }
I20240503 08:11:12.094818 1480 catalog_manager.cc:5484] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf reported cstate change: term changed from 1 to 2, leader changed from af90f15247b04a15a349855ad8e603a6 (127.0.230.2) to 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3). New cstate: current_term: 2 leader_uuid: "93516627678c47c6b7c9b28e78ae78cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:12.106448 1846 consensus_queue.cc:1035] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.110630 1788 consensus_queue.cc:1035] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.195492 1787 raft_consensus.cc:979] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6: : Instructing follower 93516627678c47c6b7c9b28e78ae78cf to start an election
I20240503 08:11:12.195681 1787 raft_consensus.cc:1067] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Signalling peer 93516627678c47c6b7c9b28e78ae78cf to start an election
I20240503 08:11:12.196272 1733 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8724626d99334661a58a321b6c178468"
dest_uuid: "93516627678c47c6b7c9b28e78ae78cf"
from {username='slave'} at 127.0.0.1:60228
I20240503 08:11:12.196491 1733 raft_consensus.cc:482] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:12.196588 1733 raft_consensus.cc:3037] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:12.198457 1733 raft_consensus.cc:504] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } }
I20240503 08:11:12.199030 1733 leader_election.cc:288] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [CANDIDATE]: Term 2 election: Requested vote from peers af90f15247b04a15a349855ad8e603a6 (127.0.230.2:35187), bba5e9184a07463a9c52576421281521 (127.0.230.1:40443)
I20240503 08:11:12.199489 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8724626d99334661a58a321b6c178468" candidate_uuid: "93516627678c47c6b7c9b28e78ae78cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "af90f15247b04a15a349855ad8e603a6"
I20240503 08:11:12.199514 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8724626d99334661a58a321b6c178468" candidate_uuid: "93516627678c47c6b7c9b28e78ae78cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bba5e9184a07463a9c52576421281521"
I20240503 08:11:12.199668 1659 raft_consensus.cc:3032] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:12.199673 1583 raft_consensus.cc:3037] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:12.199779 1659 raft_consensus.cc:729] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af90f15247b04a15a349855ad8e603a6, State: Running, Role: LEADER
I20240503 08:11:12.200042 1659 consensus_queue.cc:260] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } }
I20240503 08:11:12.200349 1659 raft_consensus.cc:3037] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:12.201764 1583 raft_consensus.cc:2454] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93516627678c47c6b7c9b28e78ae78cf in term 2.
I20240503 08:11:12.202100 1696 leader_election.cc:302] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93516627678c47c6b7c9b28e78ae78cf, bba5e9184a07463a9c52576421281521; no voters:
I20240503 08:11:12.202369 1659 raft_consensus.cc:2454] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93516627678c47c6b7c9b28e78ae78cf in term 2.
I20240503 08:11:12.202371 1786 raft_consensus.cc:2781] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:12.202575 1786 raft_consensus.cc:686] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 2 LEADER]: Becoming Leader. State: Replica: 93516627678c47c6b7c9b28e78ae78cf, State: Running, Role: LEADER
I20240503 08:11:12.202821 1786 consensus_queue.cc:237] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } }
I20240503 08:11:12.204905 1480 catalog_manager.cc:5484] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf reported cstate change: term changed from 1 to 2, leader changed from af90f15247b04a15a349855ad8e603a6 (127.0.230.2) to 93516627678c47c6b7c9b28e78ae78cf (127.0.230.3). New cstate: current_term: 2 leader_uuid: "93516627678c47c6b7c9b28e78ae78cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.329505 1803 consensus_queue.cc:1035] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.333798 1802 consensus_queue.cc:1035] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.520906 1732 raft_consensus.cc:1261] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf [term 2 FOLLOWER]: Refusing update from remote peer bba5e9184a07463a9c52576421281521: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:11:12.521622 1788 consensus_queue.cc:1035] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:12.525032 1659 raft_consensus.cc:1261] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Refusing update from remote peer bba5e9184a07463a9c52576421281521: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:11:12.525575 1817 consensus_queue.cc:1035] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:12.527594 1659 raft_consensus.cc:1261] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Refusing update from remote peer 93516627678c47c6b7c9b28e78ae78cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:11:12.528434 1786 consensus_queue.cc:1035] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:12.531481 1584 raft_consensus.cc:1261] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Refusing update from remote peer 93516627678c47c6b7c9b28e78ae78cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:11:12.532258 1786 consensus_queue.cc:1035] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:12.709898 1584 raft_consensus.cc:1261] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Refusing update from remote peer 93516627678c47c6b7c9b28e78ae78cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:11:12.710417 1786 consensus_queue.cc:1035] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:12.713179 1659 raft_consensus.cc:1261] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Refusing update from remote peer 93516627678c47c6b7c9b28e78ae78cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:11:12.713660 1919 consensus_queue.cc:1035] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:12.934796 1905 heartbeater.cc:503] Master 127.0.230.62:35749 was elected leader, sending a full tablet report...
I20240503 08:11:13.961593 1536 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:13.962908 1536 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:13.963068 1536 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:15.963429 1536 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:15.964761 1536 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:15.964926 1536 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:16.963169 1658 consensus_queue.cc:237] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } attrs { replace: true } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } }
I20240503 08:11:16.969403 1732 raft_consensus.cc:1261] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Refusing update from remote peer af90f15247b04a15a349855ad8e603a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:16.969861 1584 raft_consensus.cc:1261] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Refusing update from remote peer af90f15247b04a15a349855ad8e603a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:16.970304 1803 consensus_queue.cc:1035] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:16.970959 1931 consensus_queue.cc:1035] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:11:16.977149 1625 consensus_peers.cc:485] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 -> Peer f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4:39179): Couldn't send request to peer f1e919f7c4cb410dbcfb5970d30066bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c01a74d81a924520873129b0021e6253. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:16.977355 1803 raft_consensus.cc:2932] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } attrs { replace: true } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
I20240503 08:11:16.977905 1584 raft_consensus.cc:2932] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } attrs { replace: true } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
I20240503 08:11:16.978605 1732 raft_consensus.cc:2932] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } attrs { replace: true } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
I20240503 08:11:16.982353 1481 catalog_manager.cc:5484] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 reported cstate change: config changed from index -1 to 2, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New cstate: current_term: 1 leader_uuid: "af90f15247b04a15a349855ad8e603a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:16.992322 1584 consensus_queue.cc:237] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } attrs { replace: true } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } }
I20240503 08:11:16.994451 1658 raft_consensus.cc:1261] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Refusing update from remote peer bba5e9184a07463a9c52576421281521: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:16.994794 1732 raft_consensus.cc:1261] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Refusing update from remote peer bba5e9184a07463a9c52576421281521: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:16.995059 1942 consensus_queue.cc:1035] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:16.995409 1788 consensus_queue.cc:1035] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:16.998440 1788 raft_consensus.cc:2932] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } attrs { replace: true } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
I20240503 08:11:16.999794 1732 raft_consensus.cc:2932] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } attrs { replace: true } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
I20240503 08:11:17.000761 1658 raft_consensus.cc:2932] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } attrs { replace: true } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
W20240503 08:11:17.001508 1550 consensus_peers.cc:485] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 -> Peer f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4:39179): Couldn't send request to peer f1e919f7c4cb410dbcfb5970d30066bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fefde46bada54868808a8c3eca2c730c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:17.004906 1480 catalog_manager.cc:5484] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf reported cstate change: config changed from index -1 to 2, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New cstate: current_term: 1 leader_uuid: "bba5e9184a07463a9c52576421281521" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } attrs { replace: true } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
I20240503 08:11:17.013715 1732 consensus_queue.cc:237] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } attrs { replace: true } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } }
I20240503 08:11:17.016500 1584 raft_consensus.cc:1261] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Refusing update from remote peer 93516627678c47c6b7c9b28e78ae78cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:11:17.016716 1658 raft_consensus.cc:1261] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Refusing update from remote peer 93516627678c47c6b7c9b28e78ae78cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:11:17.017151 1949 consensus_queue.cc:1035] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:17.017525 1924 consensus_queue.cc:1035] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:17.020110 1949 raft_consensus.cc:2932] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } attrs { replace: true } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
I20240503 08:11:17.020712 1658 raft_consensus.cc:2932] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } attrs { replace: true } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
W20240503 08:11:17.023862 1699 consensus_peers.cc:485] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf -> Peer f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4:39179): Couldn't send request to peer f1e919f7c4cb410dbcfb5970d30066bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41187e251c5f43e387e077c637891fe9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:17.025506 1479 catalog_manager.cc:5484] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 reported cstate change: config changed from index -1 to 3, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New cstate: current_term: 2 leader_uuid: "93516627678c47c6b7c9b28e78ae78cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } attrs { replace: true } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
I20240503 08:11:17.027238 1732 consensus_queue.cc:237] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } attrs { replace: true } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } }
W20240503 08:11:17.029899 1699 consensus_peers.cc:485] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf -> Peer f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4:39179): Couldn't send request to peer f1e919f7c4cb410dbcfb5970d30066bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8724626d99334661a58a321b6c178468. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:17.030576 1658 raft_consensus.cc:1261] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Refusing update from remote peer 93516627678c47c6b7c9b28e78ae78cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:11:17.030994 1924 consensus_queue.cc:1035] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:17.034684 1924 raft_consensus.cc:2932] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } attrs { replace: true } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
I20240503 08:11:17.035260 1658 raft_consensus.cc:2932] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } attrs { replace: true } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
I20240503 08:11:17.036942 1584 raft_consensus.cc:2932] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } attrs { replace: true } } peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
I20240503 08:11:17.040550 1480 catalog_manager.cc:5484] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 reported cstate change: config changed from index -1 to 3, NON_VOTER f1e919f7c4cb410dbcfb5970d30066bb (127.0.230.4) added. New cstate: current_term: 2 leader_uuid: "93516627678c47c6b7c9b28e78ae78cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93516627678c47c6b7c9b28e78ae78cf" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44869 } } peers { permanent_uuid: "af90f15247b04a15a349855ad8e603a6" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 35187 } } peers { permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } attrs { replace: true } } peers { permanent_uuid: "f1e919f7c4cb410dbcfb5970d30066bb" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39179 } attrs { promote: true } } }
I20240503 08:11:17.042140 1584 raft_consensus.cc:1261] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Refusing update from remote peer 93516627678c47c6b7c9b28e78ae78cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:11:17.044216 1954 consensus_queue.cc:1035] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba5e9184a07463a9c52576421281521" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 40443 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:17.044410 920 tablet_server.cc:178] TabletServer@127.0.230.1:0 shutting down...
W20240503 08:11:17.046396 1696 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20240503 08:11:17.054919 1696 consensus_peers.cc:485] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf -> Peer bba5e9184a07463a9c52576421281521 (127.0.230.1:40443): Couldn't send request to peer bba5e9184a07463a9c52576421281521. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:17.055873 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240503 08:11:17.055976 1530 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:11:17.056073 1530 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 08:11:17.056123 920 tablet_replica.cc:330] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521: stopping tablet replica
I20240503 08:11:17.056310 920 raft_consensus.cc:2229] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.056537 920 raft_consensus.cc:2258] T c01a74d81a924520873129b0021e6253 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.057228 920 tablet_replica.cc:330] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521: stopping tablet replica
I20240503 08:11:17.057370 920 raft_consensus.cc:2229] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.057510 920 raft_consensus.cc:2258] T 8724626d99334661a58a321b6c178468 P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.057994 920 tablet_replica.cc:330] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521: stopping tablet replica
I20240503 08:11:17.058137 920 raft_consensus.cc:2229] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:17.058408 920 raft_consensus.cc:2258] T f7607f2c286d4ac29f373bd90f97b5d8 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.058853 920 tablet_replica.cc:330] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521: stopping tablet replica
I20240503 08:11:17.059020 920 raft_consensus.cc:2229] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.059175 920 raft_consensus.cc:2258] T 1ff28f2d0e2a4554ab9dd5240b87d83a P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.059795 920 tablet_replica.cc:330] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521: stopping tablet replica
I20240503 08:11:17.059947 920 raft_consensus.cc:2229] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.060164 920 raft_consensus.cc:2258] T 41187e251c5f43e387e077c637891fe9 P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.060926 920 tablet_replica.cc:330] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521: stopping tablet replica
I20240503 08:11:17.061105 920 raft_consensus.cc:2229] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.061250 920 raft_consensus.cc:2258] T dab1033f8b31480c9bbdd9508656b0d0 P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.061766 920 tablet_replica.cc:330] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521: stopping tablet replica
I20240503 08:11:17.061919 920 raft_consensus.cc:2229] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:11:17.062173 920 raft_consensus.cc:2258] T 847daf4866574864a071399c43a89f1c P bba5e9184a07463a9c52576421281521 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.062635 920 tablet_replica.cc:330] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521: stopping tablet replica
I20240503 08:11:17.062798 920 raft_consensus.cc:2229] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:17.063092 920 raft_consensus.cc:2258] T fefde46bada54868808a8c3eca2c730c P bba5e9184a07463a9c52576421281521 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 08:11:17.063750 1622 consensus_peers.cc:485] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 -> Peer bba5e9184a07463a9c52576421281521 (127.0.230.1:40443): Couldn't send request to peer bba5e9184a07463a9c52576421281521. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:17.079916 920 tablet_server.cc:194] TabletServer@127.0.230.1:0 shutdown complete.
I20240503 08:11:17.084259 920 tablet_server.cc:178] TabletServer@127.0.230.2:0 shutting down...
W20240503 08:11:17.090157 1530 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:11:17.090283 1530 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 08:11:17.092693 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:17.092926 920 tablet_replica.cc:330] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6: stopping tablet replica
I20240503 08:11:17.093173 920 raft_consensus.cc:2229] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.093369 920 raft_consensus.cc:2258] T 847daf4866574864a071399c43a89f1c P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.093968 920 tablet_replica.cc:330] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6: stopping tablet replica
I20240503 08:11:17.094118 920 raft_consensus.cc:2229] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:17.094439 920 raft_consensus.cc:2258] T 1ff28f2d0e2a4554ab9dd5240b87d83a P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.094949 920 tablet_replica.cc:330] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6: stopping tablet replica
I20240503 08:11:17.095115 920 raft_consensus.cc:2229] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:17.095436 920 raft_consensus.cc:2258] T c01a74d81a924520873129b0021e6253 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.096047 920 tablet_replica.cc:330] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6: stopping tablet replica
I20240503 08:11:17.096184 920 raft_consensus.cc:2229] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.096366 920 raft_consensus.cc:2258] T 8724626d99334661a58a321b6c178468 P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.096910 920 tablet_replica.cc:330] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6: stopping tablet replica
I20240503 08:11:17.097054 920 raft_consensus.cc:2229] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.097183 920 raft_consensus.cc:2258] T f7607f2c286d4ac29f373bd90f97b5d8 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.097580 920 tablet_replica.cc:330] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6: stopping tablet replica
I20240503 08:11:17.097714 920 raft_consensus.cc:2229] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.097887 920 raft_consensus.cc:2258] T fefde46bada54868808a8c3eca2c730c P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.098477 920 tablet_replica.cc:330] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6: stopping tablet replica
I20240503 08:11:17.098631 920 raft_consensus.cc:2229] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:17.098865 920 raft_consensus.cc:2258] T dab1033f8b31480c9bbdd9508656b0d0 P af90f15247b04a15a349855ad8e603a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.099262 920 tablet_replica.cc:330] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6: stopping tablet replica
I20240503 08:11:17.099436 920 raft_consensus.cc:2229] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.099628 920 raft_consensus.cc:2258] T 41187e251c5f43e387e077c637891fe9 P af90f15247b04a15a349855ad8e603a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.105072 920 tablet_server.cc:194] TabletServer@127.0.230.2:0 shutdown complete.
I20240503 08:11:17.109319 920 tablet_server.cc:178] TabletServer@127.0.230.3:0 shutting down...
W20240503 08:11:17.112191 1530 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:11:17.112320 1530 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:11:17.114972 1530 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:11:17.115075 1530 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 08:11:17.117287 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:17.117489 920 tablet_replica.cc:330] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf: stopping tablet replica
I20240503 08:11:17.117652 920 raft_consensus.cc:2229] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.117826 920 raft_consensus.cc:2258] T 847daf4866574864a071399c43a89f1c P 93516627678c47c6b7c9b28e78ae78cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.118340 920 tablet_replica.cc:330] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf: stopping tablet replica
I20240503 08:11:17.118494 920 raft_consensus.cc:2229] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.118659 920 raft_consensus.cc:2258] T fefde46bada54868808a8c3eca2c730c P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.119220 920 tablet_replica.cc:330] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf: stopping tablet replica
I20240503 08:11:17.119354 920 raft_consensus.cc:2229] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:11:17.119704 920 raft_consensus.cc:2258] T 8724626d99334661a58a321b6c178468 P 93516627678c47c6b7c9b28e78ae78cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.120230 920 tablet_replica.cc:330] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf: stopping tablet replica
I20240503 08:11:17.120371 920 raft_consensus.cc:2229] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.120496 920 raft_consensus.cc:2258] T f7607f2c286d4ac29f373bd90f97b5d8 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.120885 920 tablet_replica.cc:330] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf: stopping tablet replica
I20240503 08:11:17.121009 920 raft_consensus.cc:2229] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.121145 920 raft_consensus.cc:2258] T 1ff28f2d0e2a4554ab9dd5240b87d83a P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.121531 920 tablet_replica.cc:330] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf: stopping tablet replica
I20240503 08:11:17.121665 920 raft_consensus.cc:2229] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.121814 920 raft_consensus.cc:2258] T c01a74d81a924520873129b0021e6253 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.122326 920 tablet_replica.cc:330] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf: stopping tablet replica
I20240503 08:11:17.122445 920 raft_consensus.cc:2229] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:11:17.122746 920 raft_consensus.cc:2258] T 41187e251c5f43e387e077c637891fe9 P 93516627678c47c6b7c9b28e78ae78cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.123291 920 tablet_replica.cc:330] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf: stopping tablet replica
I20240503 08:11:17.123472 920 raft_consensus.cc:2229] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.123607 920 raft_consensus.cc:2258] T dab1033f8b31480c9bbdd9508656b0d0 P 93516627678c47c6b7c9b28e78ae78cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.128412 920 tablet_server.cc:194] TabletServer@127.0.230.3:0 shutdown complete.
I20240503 08:11:17.132109 920 tablet_server.cc:178] TabletServer@127.0.230.4:0 shutting down...
I20240503 08:11:17.138870 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:17.151438 920 tablet_server.cc:194] TabletServer@127.0.230.4:0 shutdown complete.
I20240503 08:11:17.153914 920 master.cc:537] Master@127.0.230.62:35749 shutting down...
I20240503 08:11:17.160661 920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:17.160883 920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.160995 920 tablet_replica.cc:330] T 00000000000000000000000000000000 P 34991c958d30454fb341102af7fa8fcc: stopping tablet replica
I20240503 08:11:17.164538 920 master.cc:552] Master@127.0.230.62:35749 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7284 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240503 08:11:17.180001 920 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.230.62:39249,127.0.230.61:46589,127.0.230.60:42427
I20240503 08:11:17.180622 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:17.183326 1961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.183939 1962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.184741 1964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.185386 920 server_base.cc:1034] running on GCE node
I20240503 08:11:17.185618 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:17.185674 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:17.185724 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723877185698 us; error 0 us; skew 500 ppm
I20240503 08:11:17.185856 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:17.187132 920 webserver.cc:460] Webserver started at http://127.0.230.62:41771/ using document root <none> and password file <none>
I20240503 08:11:17.187342 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.187486 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.187609 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.188217 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-0-root/instance:
uuid: "8978f722ce2249c28789177d4a8197a5"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-2jf9"
I20240503 08:11:17.190569 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 08:11:17.192123 1969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.192438 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:17.192543 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-0-root
uuid: "8978f722ce2249c28789177d4a8197a5"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-2jf9"
I20240503 08:11:17.192667 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.205303 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.205899 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.218291 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.62:39249
I20240503 08:11:17.218305 2020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.62:39249 every 8 connection(s)
I20240503 08:11:17.219674 2021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.219692 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:17.222246 2023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.222959 2024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.223380 2026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.223436 2021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:17.223567 920 server_base.cc:1034] running on GCE node
I20240503 08:11:17.223901 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:17.223973 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:17.224046 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723877224011 us; error 0 us; skew 500 ppm
I20240503 08:11:17.224211 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:17.226017 920 webserver.cc:460] Webserver started at http://127.0.230.61:33243/ using document root <none> and password file <none>
I20240503 08:11:17.226327 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.226460 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.226611 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.227308 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-1-root/instance:
uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-2jf9"
I20240503 08:11:17.230432 920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:17.231592 2021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:17.232434 2033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.232839 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:17.232950 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-1-root
uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-2jf9"
I20240503 08:11:17.233088 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240503 08:11:17.233601 2021 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.230.61:46589: Network error: Client connection negotiation failed: client connection to 127.0.230.61:46589: connect: Connection refused (error 111)
I20240503 08:11:17.244261 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.244805 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.252851 2021 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } attempt: 1
W20240503 08:11:17.254822 2021 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.230.61:46589: Network error: Client connection negotiation failed: client connection to 127.0.230.61:46589: connect: Connection refused (error 111)
I20240503 08:11:17.256740 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.61:46589
I20240503 08:11:17.256795 2085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.61:46589 every 8 connection(s)
I20240503 08:11:17.257994 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:11:17.258077 2086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:11:17.260700 2088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.261467 2086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } has no permanent_uuid. Determining permanent_uuid...
W20240503 08:11:17.261706 2089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.262097 2091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.262822 920 server_base.cc:1034] running on GCE node
I20240503 08:11:17.263077 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:17.263149 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:17.263234 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723877263186 us; error 0 us; skew 500 ppm
I20240503 08:11:17.263429 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:17.264734 920 webserver.cc:460] Webserver started at http://127.0.230.60:36525/ using document root <none> and password file <none>
I20240503 08:11:17.264950 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.265044 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.265169 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.265830 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-2-root/instance:
uuid: "b920f7f488a7474b80063b7a4a5c9ebc"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-2jf9"
I20240503 08:11:17.267983 2086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:17.268471 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20240503 08:11:17.270287 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.270664 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 08:11:17.270802 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-2-root
uuid: "b920f7f488a7474b80063b7a4a5c9ebc"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-2jf9"
I20240503 08:11:17.270934 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.274493 2086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } has no permanent_uuid. Determining permanent_uuid...
W20240503 08:11:17.276300 2086 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.230.60:42427: Network error: Client connection negotiation failed: client connection to 127.0.230.60:42427: connect: Connection refused (error 111)
I20240503 08:11:17.277740 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.278264 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.289173 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.60:42427
I20240503 08:11:17.289207 2150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.60:42427 every 8 connection(s)
I20240503 08:11:17.290019 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:11:17.290485 2151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.292829 2151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:17.298457 2151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:17.303177 2151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:17.306484 2086 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } attempt: 1
I20240503 08:11:17.311302 2151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc: Bootstrap starting.
I20240503 08:11:17.313005 2151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.314455 2086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9: Bootstrap starting.
I20240503 08:11:17.314651 2151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc: No bootstrap required, opened a new log
I20240503 08:11:17.315295 2151 raft_consensus.cc:348] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:17.315513 2151 raft_consensus.cc:374] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.315593 2151 raft_consensus.cc:729] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b920f7f488a7474b80063b7a4a5c9ebc, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.315752 2086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.315835 2151 consensus_queue.cc:260] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:17.316493 2157 sys_catalog.cc:455] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:17.316782 2157 sys_catalog.cc:458] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:17.316854 2151 sys_catalog.cc:564] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:17.317440 2086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9: No bootstrap required, opened a new log
I20240503 08:11:17.318064 2086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:17.318257 2086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.318347 2086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bf75fe6a3364e4aa0de3cdb7a3450d9, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.318616 2086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:17.319192 2164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:17.319360 2086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:17.319511 2164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [sys.catalog]: This master's current role is: FOLLOWER
W20240503 08:11:17.322299 2173 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:11:17.322404 2173 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:11:17.323027 2021 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } attempt: 2
W20240503 08:11:17.324529 2181 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:11:17.324621 2181 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:11:17.328212 2021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:11:17.335192 2021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5: Bootstrap starting.
I20240503 08:11:17.336465 2021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.338055 2021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5: No bootstrap required, opened a new log
I20240503 08:11:17.338626 2021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:17.338821 2021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.338891 2021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8978f722ce2249c28789177d4a8197a5, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.339102 2021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:17.339665 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:17.339836 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:17.339891 2021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:17.343350 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240503 08:11:17.343472 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 08:11:17.344148 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:17.344341 2194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:11:17.344424 2194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 08:11:17.347319 2195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.348057 2196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.348861 2198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.349345 920 server_base.cc:1034] running on GCE node
I20240503 08:11:17.349607 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:17.349678 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:17.349716 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723877349704 us; error 0 us; skew 500 ppm
I20240503 08:11:17.349850 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:17.351056 920 webserver.cc:460] Webserver started at http://127.0.230.1:39261/ using document root <none> and password file <none>
I20240503 08:11:17.351269 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.351338 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.351491 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.352104 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-0-root/instance:
uuid: "a2a5b2f4ac584fb38dc41bbba7c93799"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-2jf9"
I20240503 08:11:17.354465 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:17.356029 2203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.356355 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:11:17.356474 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-0-root
uuid: "a2a5b2f4ac584fb38dc41bbba7c93799"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-2jf9"
I20240503 08:11:17.356590 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.384270 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.384876 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.385468 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:17.386381 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:17.386467 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.386559 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:17.386631 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.400164 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.1:44209
I20240503 08:11:17.400194 2265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.1:44209 every 8 connection(s)
I20240503 08:11:17.403265 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:17.410056 2273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.410867 2267 heartbeater.cc:347] Connected to a master server at 127.0.230.62:39249
I20240503 08:11:17.411069 2267 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.411078 2266 heartbeater.cc:347] Connected to a master server at 127.0.230.60:42427
I20240503 08:11:17.411244 2266 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.411712 2267 heartbeater.cc:511] Master 127.0.230.62:39249 requested a full tablet report, sending...
I20240503 08:11:17.411808 2266 heartbeater.cc:511] Master 127.0.230.60:42427 requested a full tablet report, sending...
I20240503 08:11:17.413010 2116 ts_manager.cc:194] Registered new tserver with Master: a2a5b2f4ac584fb38dc41bbba7c93799 (127.0.230.1:44209)
I20240503 08:11:17.413674 1986 ts_manager.cc:194] Registered new tserver with Master: a2a5b2f4ac584fb38dc41bbba7c93799 (127.0.230.1:44209)
W20240503 08:11:17.413682 2274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.414841 2268 heartbeater.cc:347] Connected to a master server at 127.0.230.61:46589
I20240503 08:11:17.414988 2268 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.415313 2268 heartbeater.cc:511] Master 127.0.230.61:46589 requested a full tablet report, sending...
I20240503 08:11:17.415685 920 server_base.cc:1034] running on GCE node
I20240503 08:11:17.416208 2051 ts_manager.cc:194] Registered new tserver with Master: a2a5b2f4ac584fb38dc41bbba7c93799 (127.0.230.1:44209)
W20240503 08:11:17.416631 2276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.416965 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:17.417032 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:17.417100 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723877417069 us; error 0 us; skew 500 ppm
I20240503 08:11:17.417248 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:17.418486 920 webserver.cc:460] Webserver started at http://127.0.230.2:39265/ using document root <none> and password file <none>
I20240503 08:11:17.418691 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.418780 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.418915 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.419649 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-1-root/instance:
uuid: "5f585f30d1e84f6c92787f7b3932b01f"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-2jf9"
I20240503 08:11:17.421931 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:17.423444 2281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.423803 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:17.423900 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-1-root
uuid: "5f585f30d1e84f6c92787f7b3932b01f"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-2jf9"
I20240503 08:11:17.424017 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.445093 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.445667 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.446214 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:17.447046 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:17.447108 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.447207 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:17.447252 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.461975 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.2:46319
I20240503 08:11:17.462004 2343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.2:46319 every 8 connection(s)
I20240503 08:11:17.470742 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:11:17.471987 2345 heartbeater.cc:347] Connected to a master server at 127.0.230.62:39249
I20240503 08:11:17.472097 2344 heartbeater.cc:347] Connected to a master server at 127.0.230.60:42427
I20240503 08:11:17.472223 2345 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.472240 2344 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.472882 2344 heartbeater.cc:511] Master 127.0.230.60:42427 requested a full tablet report, sending...
I20240503 08:11:17.473305 2345 heartbeater.cc:511] Master 127.0.230.62:39249 requested a full tablet report, sending...
I20240503 08:11:17.474009 2116 ts_manager.cc:194] Registered new tserver with Master: 5f585f30d1e84f6c92787f7b3932b01f (127.0.230.2:46319)
I20240503 08:11:17.474354 1986 ts_manager.cc:194] Registered new tserver with Master: 5f585f30d1e84f6c92787f7b3932b01f (127.0.230.2:46319)
I20240503 08:11:17.476693 2346 heartbeater.cc:347] Connected to a master server at 127.0.230.61:46589
I20240503 08:11:17.476985 2346 heartbeater.cc:464] Registering TS with master...
W20240503 08:11:17.477080 2352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.476740 2351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.477548 2346 heartbeater.cc:511] Master 127.0.230.61:46589 requested a full tablet report, sending...
I20240503 08:11:17.478514 2051 ts_manager.cc:194] Registered new tserver with Master: 5f585f30d1e84f6c92787f7b3932b01f (127.0.230.2:46319)
I20240503 08:11:17.478744 920 server_base.cc:1034] running on GCE node
W20240503 08:11:17.478991 2354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.479326 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:17.479416 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:17.479475 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723877479456 us; error 0 us; skew 500 ppm
I20240503 08:11:17.479641 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:17.480787 920 webserver.cc:460] Webserver started at http://127.0.230.3:39053/ using document root <none> and password file <none>
I20240503 08:11:17.480999 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.481077 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.481207 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.481796 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-2-root/instance:
uuid: "71b0ebbbce1c412a86d84e1b4947a274"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-2jf9"
I20240503 08:11:17.484123 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 08:11:17.485620 2359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.486011 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:17.486126 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-2-root
uuid: "71b0ebbbce1c412a86d84e1b4947a274"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-2jf9"
I20240503 08:11:17.486232 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723866726930-920-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.509603 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.510238 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.510871 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:17.511782 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:17.511871 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.511963 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:17.512033 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.527496 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.3:38477
I20240503 08:11:17.527551 2421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.3:38477 every 8 connection(s)
I20240503 08:11:17.542441 2422 heartbeater.cc:347] Connected to a master server at 127.0.230.60:42427
I20240503 08:11:17.542616 2422 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.543129 2422 heartbeater.cc:511] Master 127.0.230.60:42427 requested a full tablet report, sending...
I20240503 08:11:17.544310 2423 heartbeater.cc:347] Connected to a master server at 127.0.230.62:39249
I20240503 08:11:17.544375 2116 ts_manager.cc:194] Registered new tserver with Master: 71b0ebbbce1c412a86d84e1b4947a274 (127.0.230.3:38477)
I20240503 08:11:17.544533 2423 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.545027 2423 heartbeater.cc:511] Master 127.0.230.62:39249 requested a full tablet report, sending...
I20240503 08:11:17.545997 1986 ts_manager.cc:194] Registered new tserver with Master: 71b0ebbbce1c412a86d84e1b4947a274 (127.0.230.3:38477)
I20240503 08:11:17.547195 2424 heartbeater.cc:347] Connected to a master server at 127.0.230.61:46589
I20240503 08:11:17.547297 2424 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.547632 2424 heartbeater.cc:511] Master 127.0.230.61:46589 requested a full tablet report, sending...
I20240503 08:11:17.548343 2051 ts_manager.cc:194] Registered new tserver with Master: 71b0ebbbce1c412a86d84e1b4947a274 (127.0.230.3:38477)
I20240503 08:11:17.548609 920 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006842862s
I20240503 08:11:17.674651 2164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:17.674868 2164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:17.675700 2164 leader_election.cc:288] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8978f722ce2249c28789177d4a8197a5 (127.0.230.62:39249), b920f7f488a7474b80063b7a4a5c9ebc (127.0.230.60:42427)
I20240503 08:11:17.676209 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8978f722ce2249c28789177d4a8197a5" is_pre_election: true
I20240503 08:11:17.676230 2126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" is_pre_election: true
I20240503 08:11:17.676452 1996 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bf75fe6a3364e4aa0de3cdb7a3450d9 in term 0.
I20240503 08:11:17.676483 2126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bf75fe6a3364e4aa0de3cdb7a3450d9 in term 0.
I20240503 08:11:17.676878 2038 leader_election.cc:302] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3bf75fe6a3364e4aa0de3cdb7a3450d9, 8978f722ce2249c28789177d4a8197a5, b920f7f488a7474b80063b7a4a5c9ebc; no voters:
I20240503 08:11:17.677124 2164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:17.677232 2164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:17.677300 2164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.679090 2164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:17.679612 2164 leader_election.cc:288] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [CANDIDATE]: Term 1 election: Requested vote from peers 8978f722ce2249c28789177d4a8197a5 (127.0.230.62:39249), b920f7f488a7474b80063b7a4a5c9ebc (127.0.230.60:42427)
I20240503 08:11:17.679975 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8978f722ce2249c28789177d4a8197a5"
I20240503 08:11:17.680086 2126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b920f7f488a7474b80063b7a4a5c9ebc"
I20240503 08:11:17.680137 1996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.680255 2126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.682235 2126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bf75fe6a3364e4aa0de3cdb7a3450d9 in term 1.
I20240503 08:11:17.682256 1996 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bf75fe6a3364e4aa0de3cdb7a3450d9 in term 1.
I20240503 08:11:17.682593 2035 leader_election.cc:302] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3bf75fe6a3364e4aa0de3cdb7a3450d9, 8978f722ce2249c28789177d4a8197a5, b920f7f488a7474b80063b7a4a5c9ebc; no voters:
I20240503 08:11:17.682854 2164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:17.683223 2164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 1 LEADER]: Becoming Leader. State: Replica: 3bf75fe6a3364e4aa0de3cdb7a3450d9, State: Running, Role: LEADER
I20240503 08:11:17.683490 2164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:17.684707 2430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bf75fe6a3364e4aa0de3cdb7a3450d9. Latest consensus state: current_term: 1 leader_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:17.685005 2430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:17.685596 2432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:17.687011 2432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:17.689842 2126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 1 FOLLOWER]: Refusing update from remote peer 3bf75fe6a3364e4aa0de3cdb7a3450d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:17.689842 1996 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 1 FOLLOWER]: Refusing update from remote peer 3bf75fe6a3364e4aa0de3cdb7a3450d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:17.690385 2164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:17.690637 2430 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:17.692924 2157 sys_catalog.cc:455] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bf75fe6a3364e4aa0de3cdb7a3450d9. Latest consensus state: current_term: 1 leader_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:17.693217 2157 sys_catalog.cc:458] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:17.694808 2164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:17.695050 2164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:17.695407 2432 catalog_manager.cc:1347] Generated new cluster ID: 57456543c8ea47a188aaf8d227a3e252
I20240503 08:11:17.695493 2432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:17.699900 2157 sys_catalog.cc:455] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:17.700107 2157 sys_catalog.cc:458] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:17.700961 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bf75fe6a3364e4aa0de3cdb7a3450d9. Latest consensus state: current_term: 1 leader_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:17.701179 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:17.702280 2164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:17.702489 2164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:17.703483 2432 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:17.704689 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:17.704874 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:17.705646 2432 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:17.721148 2432 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9: Generated new TSK 0
I20240503 08:11:17.721479 2432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:18.323688 2173 catalog_manager.cc:1259] Loaded cluster ID: 57456543c8ea47a188aaf8d227a3e252
I20240503 08:11:18.323792 2173 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc: loading cluster ID for follower catalog manager: success
I20240503 08:11:18.325695 2173 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc: acquiring CA information for follower catalog manager: success
I20240503 08:11:18.326623 2173 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:11:18.345614 2194 catalog_manager.cc:1259] Loaded cluster ID: 57456543c8ea47a188aaf8d227a3e252
I20240503 08:11:18.345723 2194 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5: loading cluster ID for follower catalog manager: success
I20240503 08:11:18.347541 2194 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5: acquiring CA information for follower catalog manager: success
I20240503 08:11:18.348487 2194 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:11:18.418367 2051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42014
I20240503 08:11:18.480455 2051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42020
I20240503 08:11:18.550333 2051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42024
I20240503 08:11:19.323527 2180 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:19.419531 2268 heartbeater.cc:503] Master 127.0.230.61:46589 was elected leader, sending a full tablet report...
I20240503 08:11:19.481526 2346 heartbeater.cc:503] Master 127.0.230.61:46589 was elected leader, sending a full tablet report...
I20240503 08:11:19.551494 2424 heartbeater.cc:503] Master 127.0.230.61:46589 was elected leader, sending a full tablet report...
I20240503 08:11:19.807590 920 test_util.cc:268] Using random seed: 1136552531
I20240503 08:11:19.829352 2051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42038:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 08:11:19.865520 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 3f07732e162b469a9508d4d66a78005c (DEFAULT_TABLE table=test-workload [id=d7870dea3b2b48e49a64e00cb77201d9]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 08:11:19.866153 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f07732e162b469a9508d4d66a78005c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.866565 2309 tablet_service.cc:1467] Processing CreateTablet for tablet f5aae0831bea44f68f8fd0e8ba55fee3 (DEFAULT_TABLE table=test-workload [id=d7870dea3b2b48e49a64e00cb77201d9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 08:11:19.866968 2309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5aae0831bea44f68f8fd0e8ba55fee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.871001 2468 tablet_bootstrap.cc:492] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274: Bootstrap starting.
I20240503 08:11:19.873036 2468 tablet_bootstrap.cc:654] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.873715 2469 tablet_bootstrap.cc:492] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f: Bootstrap starting.
I20240503 08:11:19.873762 2231 tablet_service.cc:1467] Processing CreateTablet for tablet a6d77e5cd1324081bae53c021dcede48 (DEFAULT_TABLE table=test-workload [id=d7870dea3b2b48e49a64e00cb77201d9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 08:11:19.874297 2231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6d77e5cd1324081bae53c021dcede48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:19.875365 2469 tablet_bootstrap.cc:654] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.875474 2468 tablet_bootstrap.cc:492] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274: No bootstrap required, opened a new log
I20240503 08:11:19.875634 2468 ts_tablet_manager.cc:1367] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:11:19.876441 2468 raft_consensus.cc:348] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b0ebbbce1c412a86d84e1b4947a274" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38477 } }
I20240503 08:11:19.876659 2468 raft_consensus.cc:374] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.876782 2468 raft_consensus.cc:729] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b0ebbbce1c412a86d84e1b4947a274, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.877027 2468 consensus_queue.cc:260] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b0ebbbce1c412a86d84e1b4947a274" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38477 } }
I20240503 08:11:19.877195 2469 tablet_bootstrap.cc:492] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f: No bootstrap required, opened a new log
I20240503 08:11:19.877218 2468 raft_consensus.cc:388] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:19.877368 2469 ts_tablet_manager.cc:1367] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:19.877372 2468 raft_consensus.cc:482] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:19.877529 2468 raft_consensus.cc:3037] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.878129 2469 raft_consensus.cc:348] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f585f30d1e84f6c92787f7b3932b01f" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 46319 } }
I20240503 08:11:19.878297 2469 raft_consensus.cc:374] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.878410 2469 raft_consensus.cc:729] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f585f30d1e84f6c92787f7b3932b01f, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.878696 2469 consensus_queue.cc:260] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f585f30d1e84f6c92787f7b3932b01f" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 46319 } }
I20240503 08:11:19.878906 2469 raft_consensus.cc:388] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:19.878989 2469 raft_consensus.cc:482] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:19.879112 2469 raft_consensus.cc:3037] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.880441 2468 raft_consensus.cc:504] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b0ebbbce1c412a86d84e1b4947a274" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38477 } }
I20240503 08:11:19.880697 2468 leader_election.cc:302] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71b0ebbbce1c412a86d84e1b4947a274; no voters:
I20240503 08:11:19.881148 2468 leader_election.cc:288] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:19.881345 2473 raft_consensus.cc:2781] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:19.882354 2469 raft_consensus.cc:504] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f585f30d1e84f6c92787f7b3932b01f" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 46319 } }
I20240503 08:11:19.882661 2469 leader_election.cc:302] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f585f30d1e84f6c92787f7b3932b01f; no voters:
I20240503 08:11:19.888861 2472 tablet_bootstrap.cc:492] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799: Bootstrap starting.
I20240503 08:11:19.889526 2468 ts_tablet_manager.cc:1398] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20240503 08:11:19.889533 2473 raft_consensus.cc:686] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [term 1 LEADER]: Becoming Leader. State: Replica: 71b0ebbbce1c412a86d84e1b4947a274, State: Running, Role: LEADER
I20240503 08:11:19.890075 2473 consensus_queue.cc:237] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b0ebbbce1c412a86d84e1b4947a274" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38477 } }
I20240503 08:11:19.891175 2469 leader_election.cc:288] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:19.891332 2476 raft_consensus.cc:2781] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:19.891855 2472 tablet_bootstrap.cc:654] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:19.894258 2051 catalog_manager.cc:5484] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71b0ebbbce1c412a86d84e1b4947a274 (127.0.230.3). New cstate: current_term: 1 leader_uuid: "71b0ebbbce1c412a86d84e1b4947a274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b0ebbbce1c412a86d84e1b4947a274" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38477 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:19.904708 2476 raft_consensus.cc:686] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [term 1 LEADER]: Becoming Leader. State: Replica: 5f585f30d1e84f6c92787f7b3932b01f, State: Running, Role: LEADER
I20240503 08:11:19.905097 2476 consensus_queue.cc:237] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f585f30d1e84f6c92787f7b3932b01f" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 46319 } }
I20240503 08:11:19.906036 2469 ts_tablet_manager.cc:1398] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f: Time spent starting tablet: real 0.029s user 0.011s sys 0.014s
I20240503 08:11:19.907102 2472 tablet_bootstrap.cc:492] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799: No bootstrap required, opened a new log
I20240503 08:11:19.907265 2472 ts_tablet_manager.cc:1367] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.003s
I20240503 08:11:19.908140 2472 raft_consensus.cc:348] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a5b2f4ac584fb38dc41bbba7c93799" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 44209 } }
I20240503 08:11:19.908177 2051 catalog_manager.cc:5484] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f585f30d1e84f6c92787f7b3932b01f (127.0.230.2). New cstate: current_term: 1 leader_uuid: "5f585f30d1e84f6c92787f7b3932b01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f585f30d1e84f6c92787f7b3932b01f" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 46319 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:19.908320 2472 raft_consensus.cc:374] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:19.908412 2472 raft_consensus.cc:729] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a5b2f4ac584fb38dc41bbba7c93799, State: Initialized, Role: FOLLOWER
I20240503 08:11:19.908624 2472 consensus_queue.cc:260] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a5b2f4ac584fb38dc41bbba7c93799" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 44209 } }
I20240503 08:11:19.908813 2472 raft_consensus.cc:388] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:19.908897 2472 raft_consensus.cc:482] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:19.909031 2472 raft_consensus.cc:3037] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:19.912037 2472 raft_consensus.cc:504] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a5b2f4ac584fb38dc41bbba7c93799" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 44209 } }
I20240503 08:11:19.912284 2472 leader_election.cc:302] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a5b2f4ac584fb38dc41bbba7c93799; no voters:
I20240503 08:11:19.913827 2472 leader_election.cc:288] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:19.913992 2484 raft_consensus.cc:2781] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:19.914544 2472 ts_tablet_manager.cc:1398] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240503 08:11:19.915628 2484 raft_consensus.cc:686] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [term 1 LEADER]: Becoming Leader. State: Replica: a2a5b2f4ac584fb38dc41bbba7c93799, State: Running, Role: LEADER
I20240503 08:11:19.915923 2484 consensus_queue.cc:237] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a5b2f4ac584fb38dc41bbba7c93799" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 44209 } }
I20240503 08:11:19.918937 2050 catalog_manager.cc:5484] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a5b2f4ac584fb38dc41bbba7c93799 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "a2a5b2f4ac584fb38dc41bbba7c93799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a5b2f4ac584fb38dc41bbba7c93799" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 44209 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:19.941769 920 master.cc:537] Master@127.0.230.61:46589 shutting down...
I20240503 08:11:19.947753 920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:19.948110 920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:19.948247 920 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3bf75fe6a3364e4aa0de3cdb7a3450d9: stopping tablet replica
I20240503 08:11:19.963135 920 master.cc:552] Master@127.0.230.61:46589 shutdown complete.
W20240503 08:11:20.905112 2424 heartbeater.cc:644] Failed to heartbeat to 127.0.230.61:46589 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.230.61:46589: connect: Connection refused (error 111)
I20240503 08:11:21.426244 2493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3bf75fe6a3364e4aa0de3cdb7a3450d9)
I20240503 08:11:21.426496 2493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:21.426738 2494 raft_consensus.cc:482] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3bf75fe6a3364e4aa0de3cdb7a3450d9)
I20240503 08:11:21.426920 2494 raft_consensus.cc:504] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:21.427281 2493 leader_election.cc:288] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589), b920f7f488a7474b80063b7a4a5c9ebc (127.0.230.60:42427)
I20240503 08:11:21.427613 2494 leader_election.cc:288] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8978f722ce2249c28789177d4a8197a5 (127.0.230.62:39249), 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589)
I20240503 08:11:21.428288 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "8978f722ce2249c28789177d4a8197a5" is_pre_election: true
I20240503 08:11:21.428512 1995 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b920f7f488a7474b80063b7a4a5c9ebc in term 1.
I20240503 08:11:21.428905 2126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8978f722ce2249c28789177d4a8197a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" is_pre_election: true
I20240503 08:11:21.429111 2126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8978f722ce2249c28789177d4a8197a5 in term 1.
W20240503 08:11:21.429544 1970 leader_election.cc:334] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589): Network error: Client connection negotiation failed: client connection to 127.0.230.61:46589: connect: Connection refused (error 111)
I20240503 08:11:21.429774 1973 leader_election.cc:302] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8978f722ce2249c28789177d4a8197a5, b920f7f488a7474b80063b7a4a5c9ebc; no voters: 3bf75fe6a3364e4aa0de3cdb7a3450d9
I20240503 08:11:21.429819 2099 leader_election.cc:302] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8978f722ce2249c28789177d4a8197a5, b920f7f488a7474b80063b7a4a5c9ebc; no voters:
I20240503 08:11:21.430017 2493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240503 08:11:21.430061 2099 leader_election.cc:334] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589): Network error: Client connection negotiation failed: client connection to 127.0.230.61:46589: connect: Connection refused (error 111)
I20240503 08:11:21.430136 2493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3bf75fe6a3364e4aa0de3cdb7a3450d9)
I20240503 08:11:21.430097 2494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:11:21.430243 2493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:21.430269 2494 raft_consensus.cc:482] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3bf75fe6a3364e4aa0de3cdb7a3450d9)
I20240503 08:11:21.430398 2494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:21.475029 2494 raft_consensus.cc:504] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:21.475056 2493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:21.475935 2494 leader_election.cc:288] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [CANDIDATE]: Term 2 election: Requested vote from peers 8978f722ce2249c28789177d4a8197a5 (127.0.230.62:39249), 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589)
I20240503 08:11:21.476411 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "8978f722ce2249c28789177d4a8197a5"
I20240503 08:11:21.476581 2493 leader_election.cc:288] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [CANDIDATE]: Term 2 election: Requested vote from peers 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589), b920f7f488a7474b80063b7a4a5c9ebc (127.0.230.60:42427)
I20240503 08:11:21.476723 1995 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b920f7f488a7474b80063b7a4a5c9ebc in current term 2: Already voted for candidate 8978f722ce2249c28789177d4a8197a5 in this term.
W20240503 08:11:21.477391 1970 leader_election.cc:334] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589): Network error: Client connection negotiation failed: client connection to 127.0.230.61:46589: connect: Connection refused (error 111)
W20240503 08:11:21.477560 2099 leader_election.cc:334] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589): Network error: Client connection negotiation failed: client connection to 127.0.230.61:46589: connect: Connection refused (error 111)
I20240503 08:11:21.477681 2099 leader_election.cc:302] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b920f7f488a7474b80063b7a4a5c9ebc; no voters: 3bf75fe6a3364e4aa0de3cdb7a3450d9, 8978f722ce2249c28789177d4a8197a5
I20240503 08:11:21.477664 2126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8978f722ce2249c28789177d4a8197a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "b920f7f488a7474b80063b7a4a5c9ebc"
I20240503 08:11:21.477909 2126 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8978f722ce2249c28789177d4a8197a5 in current term 2: Already voted for candidate b920f7f488a7474b80063b7a4a5c9ebc in this term.
I20240503 08:11:21.478060 2494 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 08:11:21.478276 1973 leader_election.cc:302] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8978f722ce2249c28789177d4a8197a5; no voters: 3bf75fe6a3364e4aa0de3cdb7a3450d9, b920f7f488a7474b80063b7a4a5c9ebc
I20240503 08:11:21.478508 2493 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 08:11:23.419545 2505 raft_consensus.cc:482] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:23.419768 2505 raft_consensus.cc:504] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:23.420442 2505 leader_election.cc:288] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8978f722ce2249c28789177d4a8197a5 (127.0.230.62:39249), 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589)
I20240503 08:11:23.420985 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "8978f722ce2249c28789177d4a8197a5" is_pre_election: true
I20240503 08:11:23.421234 1995 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b920f7f488a7474b80063b7a4a5c9ebc in term 2.
W20240503 08:11:23.421376 2099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.230.61:46589: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240503 08:11:23.421865 2099 leader_election.cc:302] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8978f722ce2249c28789177d4a8197a5, b920f7f488a7474b80063b7a4a5c9ebc; no voters:
I20240503 08:11:23.422109 2505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 08:11:23.422237 2505 raft_consensus.cc:482] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:23.422343 2505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 2 FOLLOWER]: Advancing to term 3
W20240503 08:11:23.422575 2099 leader_election.cc:334] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589): Network error: Client connection negotiation failed: client connection to 127.0.230.61:46589: connect: Connection refused (error 111)
I20240503 08:11:23.424319 2505 raft_consensus.cc:504] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:23.424842 2505 leader_election.cc:288] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [CANDIDATE]: Term 3 election: Requested vote from peers 8978f722ce2249c28789177d4a8197a5 (127.0.230.62:39249), 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589)
I20240503 08:11:23.425467 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "8978f722ce2249c28789177d4a8197a5"
I20240503 08:11:23.425678 1995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 2 FOLLOWER]: Advancing to term 3
W20240503 08:11:23.426092 2099 leader_election.cc:334] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589): Network error: Client connection negotiation failed: client connection to 127.0.230.61:46589: connect: Connection refused (error 111)
I20240503 08:11:23.427429 1995 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b920f7f488a7474b80063b7a4a5c9ebc in term 3.
I20240503 08:11:23.427827 2099 leader_election.cc:302] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8978f722ce2249c28789177d4a8197a5, b920f7f488a7474b80063b7a4a5c9ebc; no voters: 3bf75fe6a3364e4aa0de3cdb7a3450d9
I20240503 08:11:23.428094 2505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:11:23.428484 2505 raft_consensus.cc:686] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 3 LEADER]: Becoming Leader. State: Replica: b920f7f488a7474b80063b7a4a5c9ebc, State: Running, Role: LEADER
I20240503 08:11:23.428725 2505 consensus_queue.cc:237] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } }
I20240503 08:11:23.429791 2508 sys_catalog.cc:455] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b920f7f488a7474b80063b7a4a5c9ebc. Latest consensus state: current_term: 3 leader_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:23.430047 2508 sys_catalog.cc:458] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:23.430444 2510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:23.432094 2510 catalog_manager.cc:669] Loaded metadata for table test-workload [id=d7870dea3b2b48e49a64e00cb77201d9]
I20240503 08:11:23.433403 2510 tablet_loader.cc:96] loaded metadata for tablet 3f07732e162b469a9508d4d66a78005c (table test-workload [id=d7870dea3b2b48e49a64e00cb77201d9])
I20240503 08:11:23.433710 2510 tablet_loader.cc:96] loaded metadata for tablet a6d77e5cd1324081bae53c021dcede48 (table test-workload [id=d7870dea3b2b48e49a64e00cb77201d9])
I20240503 08:11:23.433992 2510 tablet_loader.cc:96] loaded metadata for tablet f5aae0831bea44f68f8fd0e8ba55fee3 (table test-workload [id=d7870dea3b2b48e49a64e00cb77201d9])
I20240503 08:11:23.434185 2510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:23.434960 2510 catalog_manager.cc:1259] Loaded cluster ID: 57456543c8ea47a188aaf8d227a3e252
I20240503 08:11:23.435034 2510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:23.436097 2510 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:23.436864 2510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc: Loaded TSK: 0
I20240503 08:11:23.437496 2510 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240503 08:11:23.847275 2099 consensus_peers.cc:485] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc -> Peer 3bf75fe6a3364e4aa0de3cdb7a3450d9 (127.0.230.61:46589): Couldn't send request to peer 3bf75fe6a3364e4aa0de3cdb7a3450d9. Status: Network error: Client connection negotiation failed: client connection to 127.0.230.61:46589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:24.042641 1995 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 3 FOLLOWER]: Refusing update from remote peer b920f7f488a7474b80063b7a4a5c9ebc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:11:24.043243 2508 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:11:24.044991 2514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b920f7f488a7474b80063b7a4a5c9ebc. Latest consensus state: current_term: 3 leader_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:24.045204 2514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:24.046509 2516 sys_catalog.cc:455] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:24.046808 2516 sys_catalog.cc:458] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:24.047109 2514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8978f722ce2249c28789177d4a8197a5" member_type: VOTER last_known_addr { host: "127.0.230.62" port: 39249 } } peers { permanent_uuid: "3bf75fe6a3364e4aa0de3cdb7a3450d9" member_type: VOTER last_known_addr { host: "127.0.230.61" port: 46589 } } peers { permanent_uuid: "b920f7f488a7474b80063b7a4a5c9ebc" member_type: VOTER last_known_addr { host: "127.0.230.60" port: 42427 } } }
I20240503 08:11:24.047343 2514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:11:24.904428 2422 heartbeater.cc:503] Master 127.0.230.60:42427 was elected leader, sending a full tablet report...
I20240503 08:11:24.917234 2344 heartbeater.cc:503] Master 127.0.230.60:42427 was elected leader, sending a full tablet report...
I20240503 08:11:24.932724 2266 heartbeater.cc:503] Master 127.0.230.60:42427 was elected leader, sending a full tablet report...
I20240503 08:11:25.322219 2171 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:25.322351 2171 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:25.514876 920 tablet_server.cc:178] TabletServer@127.0.230.1:0 shutting down...
I20240503 08:11:25.522394 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:25.522662 920 tablet_replica.cc:330] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799: stopping tablet replica
I20240503 08:11:25.522871 920 raft_consensus.cc:2229] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:25.523056 920 raft_consensus.cc:2258] T a6d77e5cd1324081bae53c021dcede48 P a2a5b2f4ac584fb38dc41bbba7c93799 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:25.536340 920 tablet_server.cc:194] TabletServer@127.0.230.1:0 shutdown complete.
I20240503 08:11:25.539083 920 tablet_server.cc:178] TabletServer@127.0.230.2:0 shutting down...
I20240503 08:11:25.547278 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:25.547518 920 tablet_replica.cc:330] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f: stopping tablet replica
I20240503 08:11:25.547695 920 raft_consensus.cc:2229] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:25.547869 920 raft_consensus.cc:2258] T f5aae0831bea44f68f8fd0e8ba55fee3 P 5f585f30d1e84f6c92787f7b3932b01f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:25.561115 920 tablet_server.cc:194] TabletServer@127.0.230.2:0 shutdown complete.
I20240503 08:11:25.563796 920 tablet_server.cc:178] TabletServer@127.0.230.3:0 shutting down...
I20240503 08:11:25.571593 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:25.571872 920 tablet_replica.cc:330] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274: stopping tablet replica
I20240503 08:11:25.572057 920 raft_consensus.cc:2229] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:25.572217 920 raft_consensus.cc:2258] T 3f07732e162b469a9508d4d66a78005c P 71b0ebbbce1c412a86d84e1b4947a274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:25.585484 920 tablet_server.cc:194] TabletServer@127.0.230.3:0 shutdown complete.
I20240503 08:11:25.588201 920 master.cc:537] Master@127.0.230.62:39249 shutting down...
I20240503 08:11:25.595489 920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:25.595737 920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:25.595877 920 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8978f722ce2249c28789177d4a8197a5: stopping tablet replica
I20240503 08:11:25.609115 920 master.cc:552] Master@127.0.230.62:39249 shutdown complete.
I20240503 08:11:25.612798 920 master.cc:537] Master@127.0.230.60:42427 shutting down...
I20240503 08:11:25.619319 920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 3 LEADER]: Raft consensus shutting down.
I20240503 08:11:25.619678 920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:25.619787 920 tablet_replica.cc:330] T 00000000000000000000000000000000 P b920f7f488a7474b80063b7a4a5c9ebc: stopping tablet replica
I20240503 08:11:25.632898 920 master.cc:552] Master@127.0.230.60:42427 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (8464 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240503 08:11:25.644338 920 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.230.62:45567
I20240503 08:11:25.644855 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:25.647377 2527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:25.648116 2528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:25.648692 2530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:25.649187 920 server_base.cc:1034] running on GCE node
I20240503 08:11:25.649463 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:25.649523 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:25.649587 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723885649557 us; error 0 us; skew 500 ppm
I20240503 08:11:25.649755 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:25.650938 920 webserver.cc:460] Webserver started at http://127.0.230.62:41243/ using document root <none> and password file <none>
I20240503 08:11:25.651160 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:25.651242 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:25.651376 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:25.652035 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/master-0-root/instance:
uuid: "d0ca07d9231a4c308d013354dab41b33"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-2jf9"
I20240503 08:11:25.654448 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:25.656128 2535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.656478 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:11:25.656577 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/master-0-root
uuid: "d0ca07d9231a4c308d013354dab41b33"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-2jf9"
I20240503 08:11:25.656723 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:25.666502 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:25.667045 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:25.678957 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.62:45567
I20240503 08:11:25.678978 2586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.62:45567 every 8 connection(s)
I20240503 08:11:25.680258 2587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:25.684054 2587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33: Bootstrap starting.
I20240503 08:11:25.685217 2587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:25.686604 2587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33: No bootstrap required, opened a new log
I20240503 08:11:25.687156 2587 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ca07d9231a4c308d013354dab41b33" member_type: VOTER }
I20240503 08:11:25.687280 2587 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:25.687350 2587 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0ca07d9231a4c308d013354dab41b33, State: Initialized, Role: FOLLOWER
I20240503 08:11:25.687577 2587 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ca07d9231a4c308d013354dab41b33" member_type: VOTER }
I20240503 08:11:25.687716 2587 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:25.687783 2587 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:25.687857 2587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:25.689417 2587 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ca07d9231a4c308d013354dab41b33" member_type: VOTER }
I20240503 08:11:25.689597 2587 leader_election.cc:302] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0ca07d9231a4c308d013354dab41b33; no voters:
I20240503 08:11:25.689914 2587 leader_election.cc:288] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:25.690022 2590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:25.690450 2590 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [term 1 LEADER]: Becoming Leader. State: Replica: d0ca07d9231a4c308d013354dab41b33, State: Running, Role: LEADER
I20240503 08:11:25.690752 2587 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:25.690691 2590 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ca07d9231a4c308d013354dab41b33" member_type: VOTER }
I20240503 08:11:25.691593 2591 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0ca07d9231a4c308d013354dab41b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ca07d9231a4c308d013354dab41b33" member_type: VOTER } }
I20240503 08:11:25.691637 2592 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0ca07d9231a4c308d013354dab41b33. Latest consensus state: current_term: 1 leader_uuid: "d0ca07d9231a4c308d013354dab41b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ca07d9231a4c308d013354dab41b33" member_type: VOTER } }
I20240503 08:11:25.691872 2592 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:25.691854 2591 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:25.695374 2595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:25.695443 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:11:25.696625 2595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:25.699137 2595 catalog_manager.cc:1347] Generated new cluster ID: 1127bd6b7d3f4e2781f2665293dccbce
I20240503 08:11:25.699219 2595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:25.736327 2595 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:25.737105 2595 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:25.754674 2595 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33: Generated new TSK 0
I20240503 08:11:25.754931 2595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:25.760162 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:25.762898 2608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:25.763876 2609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:25.764634 2611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:25.764917 920 server_base.cc:1034] running on GCE node
I20240503 08:11:25.765137 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:25.765202 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:25.765244 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723885765230 us; error 0 us; skew 500 ppm
I20240503 08:11:25.765364 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:25.766614 920 webserver.cc:460] Webserver started at http://127.0.230.1:44061/ using document root <none> and password file <none>
I20240503 08:11:25.766821 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:25.766886 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:25.766993 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:25.767645 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-0-root/instance:
uuid: "74177132341f434ead6ad396bd220764"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-2jf9"
I20240503 08:11:25.769968 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:25.771533 2616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.771850 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:11:25.771956 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-0-root
uuid: "74177132341f434ead6ad396bd220764"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-2jf9"
I20240503 08:11:25.772061 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:25.778421 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:25.778939 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:25.779479 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:25.780293 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:25.780377 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.780462 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:25.780534 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.793114 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.1:46715
I20240503 08:11:25.793164 2678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.1:46715 every 8 connection(s)
I20240503 08:11:25.794692 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:25.797380 2683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:25.798204 2684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:25.798506 2686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:25.799245 920 server_base.cc:1034] running on GCE node
I20240503 08:11:25.799541 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:25.799619 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:25.799700 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723885799664 us; error 0 us; skew 500 ppm
I20240503 08:11:25.799855 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:25.801205 920 webserver.cc:460] Webserver started at http://127.0.230.2:42749/ using document root <none> and password file <none>
I20240503 08:11:25.801440 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:25.801558 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:25.801702 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:25.802460 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-1-root/instance:
uuid: "4631d7d664da4054baca9a54fde1c25c"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-2jf9"
I20240503 08:11:25.804044 2679 heartbeater.cc:347] Connected to a master server at 127.0.230.62:45567
I20240503 08:11:25.804196 2679 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:25.804553 2679 heartbeater.cc:511] Master 127.0.230.62:45567 requested a full tablet report, sending...
I20240503 08:11:25.805410 2552 ts_manager.cc:194] Registered new tserver with Master: 74177132341f434ead6ad396bd220764 (127.0.230.1:46715)
I20240503 08:11:25.806375 2552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42444
I20240503 08:11:25.812630 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20240503 08:11:25.813943 2691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.814220 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:25.814307 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-1-root
uuid: "4631d7d664da4054baca9a54fde1c25c"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-2jf9"
I20240503 08:11:25.814419 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:25.842602 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:25.843119 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:25.843649 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:25.844516 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:25.844570 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.844645 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:25.844676 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.855283 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.2:41387
I20240503 08:11:25.855307 2753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.2:41387 every 8 connection(s)
I20240503 08:11:25.856639 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:25.859443 2757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:25.860179 2758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:25.860879 2760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:25.861274 920 server_base.cc:1034] running on GCE node
I20240503 08:11:25.861519 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:25.861569 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:25.861599 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723885861595 us; error 0 us; skew 500 ppm
I20240503 08:11:25.861728 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:25.862534 2754 heartbeater.cc:347] Connected to a master server at 127.0.230.62:45567
I20240503 08:11:25.862644 2754 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:25.862974 2754 heartbeater.cc:511] Master 127.0.230.62:45567 requested a full tablet report, sending...
I20240503 08:11:25.863001 920 webserver.cc:460] Webserver started at http://127.0.230.3:40197/ using document root <none> and password file <none>
I20240503 08:11:25.863171 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:25.863229 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:25.863329 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:25.863819 2552 ts_manager.cc:194] Registered new tserver with Master: 4631d7d664da4054baca9a54fde1c25c (127.0.230.2:41387)
I20240503 08:11:25.863978 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-2-root/instance:
uuid: "d09ce4a3364c4498b724195111b25e5c"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-2jf9"
I20240503 08:11:25.864547 2552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42448
I20240503 08:11:25.866170 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:25.867507 2765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.867787 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:25.867874 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-2-root
uuid: "d09ce4a3364c4498b724195111b25e5c"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-2jf9"
I20240503 08:11:25.867966 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:25.884758 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:25.885304 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:25.885807 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:25.886581 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:25.886636 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.886701 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:25.886732 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.898751 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.3:42319
I20240503 08:11:25.898783 2827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.3:42319 every 8 connection(s)
I20240503 08:11:25.904889 2828 heartbeater.cc:347] Connected to a master server at 127.0.230.62:45567
I20240503 08:11:25.905016 2828 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:25.905328 2828 heartbeater.cc:511] Master 127.0.230.62:45567 requested a full tablet report, sending...
I20240503 08:11:25.906075 2552 ts_manager.cc:194] Registered new tserver with Master: d09ce4a3364c4498b724195111b25e5c (127.0.230.3:42319)
I20240503 08:11:25.906255 920 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006707323s
I20240503 08:11:25.906780 2552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42452
I20240503 08:11:26.807571 2679 heartbeater.cc:503] Master 127.0.230.62:45567 was elected leader, sending a full tablet report...
I20240503 08:11:26.865628 2754 heartbeater.cc:503] Master 127.0.230.62:45567 was elected leader, sending a full tablet report...
I20240503 08:11:26.907977 2828 heartbeater.cc:503] Master 127.0.230.62:45567 was elected leader, sending a full tablet report...
I20240503 08:11:26.933686 920 test_util.cc:268] Using random seed: 1143678621
I20240503 08:11:26.948899 2552 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42456:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:11:26.949471 2552 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:11:26.965684 2793 tablet_service.cc:1467] Processing CreateTablet for tablet e85a6284cfa642e3b34d923737fb6e49 (DEFAULT_TABLE table=test-workload [id=994da41fea9743a7ad14e5185b4e61bb]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 08:11:26.966218 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85a6284cfa642e3b34d923737fb6e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.967192 2719 tablet_service.cc:1467] Processing CreateTablet for tablet e85a6284cfa642e3b34d923737fb6e49 (DEFAULT_TABLE table=test-workload [id=994da41fea9743a7ad14e5185b4e61bb]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 08:11:26.967345 2718 tablet_service.cc:1467] Processing CreateTablet for tablet e70cca9326a246c3992b510dcce35050 (DEFAULT_TABLE table=test-workload [id=994da41fea9743a7ad14e5185b4e61bb]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 08:11:26.967664 2719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85a6284cfa642e3b34d923737fb6e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.967871 2718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e70cca9326a246c3992b510dcce35050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.965684 2792 tablet_service.cc:1467] Processing CreateTablet for tablet e70cca9326a246c3992b510dcce35050 (DEFAULT_TABLE table=test-workload [id=994da41fea9743a7ad14e5185b4e61bb]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 08:11:26.971176 2792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e70cca9326a246c3992b510dcce35050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.971256 2848 tablet_bootstrap.cc:492] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c: Bootstrap starting.
I20240503 08:11:26.973086 2848 tablet_bootstrap.cc:654] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.975518 2848 tablet_bootstrap.cc:492] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c: No bootstrap required, opened a new log
I20240503 08:11:26.975680 2848 ts_tablet_manager.cc:1367] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:11:26.976469 2848 raft_consensus.cc:348] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } }
I20240503 08:11:26.976819 2848 raft_consensus.cc:374] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.976924 2848 raft_consensus.cc:729] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09ce4a3364c4498b724195111b25e5c, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.977197 2848 consensus_queue.cc:260] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } }
I20240503 08:11:26.977802 2644 tablet_service.cc:1467] Processing CreateTablet for tablet e85a6284cfa642e3b34d923737fb6e49 (DEFAULT_TABLE table=test-workload [id=994da41fea9743a7ad14e5185b4e61bb]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 08:11:26.977880 2643 tablet_service.cc:1467] Processing CreateTablet for tablet e70cca9326a246c3992b510dcce35050 (DEFAULT_TABLE table=test-workload [id=994da41fea9743a7ad14e5185b4e61bb]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 08:11:26.978194 2644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85a6284cfa642e3b34d923737fb6e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.978421 2643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e70cca9326a246c3992b510dcce35050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:26.978477 2849 tablet_bootstrap.cc:492] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c: Bootstrap starting.
I20240503 08:11:26.984508 2849 tablet_bootstrap.cc:654] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.985845 2848 ts_tablet_manager.cc:1398] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c: Time spent starting tablet: real 0.010s user 0.003s sys 0.007s
I20240503 08:11:26.986086 2852 tablet_bootstrap.cc:492] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764: Bootstrap starting.
I20240503 08:11:26.986253 2848 tablet_bootstrap.cc:492] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c: Bootstrap starting.
I20240503 08:11:26.987545 2852 tablet_bootstrap.cc:654] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.987848 2848 tablet_bootstrap.cc:654] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.990428 2849 tablet_bootstrap.cc:492] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c: No bootstrap required, opened a new log
I20240503 08:11:26.990548 2849 ts_tablet_manager.cc:1367] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 08:11:26.991254 2852 tablet_bootstrap.cc:492] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764: No bootstrap required, opened a new log
I20240503 08:11:26.991240 2849 raft_consensus.cc:348] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } }
I20240503 08:11:26.991415 2849 raft_consensus.cc:374] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.991472 2849 raft_consensus.cc:729] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4631d7d664da4054baca9a54fde1c25c, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.991474 2852 ts_tablet_manager.cc:1367] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20240503 08:11:26.991703 2848 tablet_bootstrap.cc:492] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c: No bootstrap required, opened a new log
I20240503 08:11:26.991667 2849 consensus_queue.cc:260] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } }
I20240503 08:11:26.991828 2848 ts_tablet_manager.cc:1367] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 08:11:26.992223 2852 raft_consensus.cc:348] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } }
I20240503 08:11:26.992365 2849 ts_tablet_manager.cc:1398] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:26.992414 2852 raft_consensus.cc:374] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.992483 2852 raft_consensus.cc:729] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74177132341f434ead6ad396bd220764, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.992460 2848 raft_consensus.cc:348] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } }
I20240503 08:11:26.992601 2849 tablet_bootstrap.cc:492] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c: Bootstrap starting.
I20240503 08:11:26.992635 2848 raft_consensus.cc:374] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.992682 2848 raft_consensus.cc:729] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09ce4a3364c4498b724195111b25e5c, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.992671 2852 consensus_queue.cc:260] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } }
I20240503 08:11:26.992923 2848 consensus_queue.cc:260] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } }
I20240503 08:11:26.993518 2852 ts_tablet_manager.cc:1398] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:26.993777 2852 tablet_bootstrap.cc:492] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764: Bootstrap starting.
I20240503 08:11:26.993970 2849 tablet_bootstrap.cc:654] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.995527 2852 tablet_bootstrap.cc:654] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:26.996312 2848 ts_tablet_manager.cc:1398] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:11:26.998332 2852 tablet_bootstrap.cc:492] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764: No bootstrap required, opened a new log
I20240503 08:11:26.998428 2852 ts_tablet_manager.cc:1367] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 08:11:26.998610 2849 tablet_bootstrap.cc:492] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c: No bootstrap required, opened a new log
I20240503 08:11:26.998718 2849 ts_tablet_manager.cc:1367] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:11:26.998944 2852 raft_consensus.cc:348] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } }
I20240503 08:11:26.999084 2852 raft_consensus.cc:374] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.999120 2852 raft_consensus.cc:729] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74177132341f434ead6ad396bd220764, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.999199 2849 raft_consensus.cc:348] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } }
I20240503 08:11:26.999346 2849 raft_consensus.cc:374] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:26.999293 2852 consensus_queue.cc:260] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } }
I20240503 08:11:26.999418 2849 raft_consensus.cc:729] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4631d7d664da4054baca9a54fde1c25c, State: Initialized, Role: FOLLOWER
I20240503 08:11:26.999604 2849 consensus_queue.cc:260] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } }
I20240503 08:11:26.999857 2852 ts_tablet_manager.cc:1398] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:11:26.999986 2849 ts_tablet_manager.cc:1398] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:27.075462 2857 raft_consensus.cc:482] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:27.075675 2857 raft_consensus.cc:504] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } }
I20240503 08:11:27.076488 2857 leader_election.cc:288] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4631d7d664da4054baca9a54fde1c25c (127.0.230.2:41387), d09ce4a3364c4498b724195111b25e5c (127.0.230.3:42319)
I20240503 08:11:27.077328 2851 raft_consensus.cc:482] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:27.077484 2851 raft_consensus.cc:504] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } }
I20240503 08:11:27.078341 2851 leader_election.cc:288] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4631d7d664da4054baca9a54fde1c25c (127.0.230.2:41387), 74177132341f434ead6ad396bd220764 (127.0.230.1:46715)
I20240503 08:11:27.084599 2729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70cca9326a246c3992b510dcce35050" candidate_uuid: "74177132341f434ead6ad396bd220764" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4631d7d664da4054baca9a54fde1c25c" is_pre_election: true
I20240503 08:11:27.084879 2729 raft_consensus.cc:2454] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74177132341f434ead6ad396bd220764 in term 0.
I20240503 08:11:27.085529 2620 leader_election.cc:302] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4631d7d664da4054baca9a54fde1c25c, 74177132341f434ead6ad396bd220764; no voters:
I20240503 08:11:27.085811 2857 raft_consensus.cc:2781] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:27.085902 2857 raft_consensus.cc:482] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:27.085955 2857 raft_consensus.cc:3037] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.089128 2803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70cca9326a246c3992b510dcce35050" candidate_uuid: "74177132341f434ead6ad396bd220764" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09ce4a3364c4498b724195111b25e5c" is_pre_election: true
I20240503 08:11:27.089416 2803 raft_consensus.cc:2454] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74177132341f434ead6ad396bd220764 in term 0.
I20240503 08:11:27.089359 2857 raft_consensus.cc:504] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } }
I20240503 08:11:27.090516 2729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70cca9326a246c3992b510dcce35050" candidate_uuid: "74177132341f434ead6ad396bd220764" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4631d7d664da4054baca9a54fde1c25c"
I20240503 08:11:27.090706 2729 raft_consensus.cc:3037] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.090745 2857 leader_election.cc:288] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [CANDIDATE]: Term 1 election: Requested vote from peers 4631d7d664da4054baca9a54fde1c25c (127.0.230.2:41387), d09ce4a3364c4498b724195111b25e5c (127.0.230.3:42319)
I20240503 08:11:27.091188 2803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70cca9326a246c3992b510dcce35050" candidate_uuid: "74177132341f434ead6ad396bd220764" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09ce4a3364c4498b724195111b25e5c"
I20240503 08:11:27.091356 2803 raft_consensus.cc:3037] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.093762 2729 raft_consensus.cc:2454] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74177132341f434ead6ad396bd220764 in term 1.
I20240503 08:11:27.094075 2803 raft_consensus.cc:2454] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74177132341f434ead6ad396bd220764 in term 1.
I20240503 08:11:27.094141 2620 leader_election.cc:302] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4631d7d664da4054baca9a54fde1c25c, 74177132341f434ead6ad396bd220764; no voters:
I20240503 08:11:27.094360 2857 raft_consensus.cc:2781] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:27.094847 2857 raft_consensus.cc:686] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 1 LEADER]: Becoming Leader. State: Replica: 74177132341f434ead6ad396bd220764, State: Running, Role: LEADER
I20240503 08:11:27.095126 2857 consensus_queue.cc:237] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } }
I20240503 08:11:27.095283 2729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85a6284cfa642e3b34d923737fb6e49" candidate_uuid: "d09ce4a3364c4498b724195111b25e5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4631d7d664da4054baca9a54fde1c25c" is_pre_election: true
I20240503 08:11:27.095467 2729 raft_consensus.cc:2454] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09ce4a3364c4498b724195111b25e5c in term 0.
I20240503 08:11:27.095809 2769 leader_election.cc:302] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4631d7d664da4054baca9a54fde1c25c, d09ce4a3364c4498b724195111b25e5c; no voters:
I20240503 08:11:27.096042 2851 raft_consensus.cc:2781] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:27.096112 2851 raft_consensus.cc:482] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:27.096174 2851 raft_consensus.cc:3037] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.097874 2552 catalog_manager.cc:5484] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74177132341f434ead6ad396bd220764 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "74177132341f434ead6ad396bd220764" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:27.098282 2851 raft_consensus.cc:504] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } }
I20240503 08:11:27.098894 2851 leader_election.cc:288] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [CANDIDATE]: Term 1 election: Requested vote from peers 4631d7d664da4054baca9a54fde1c25c (127.0.230.2:41387), 74177132341f434ead6ad396bd220764 (127.0.230.1:46715)
I20240503 08:11:27.099040 2654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85a6284cfa642e3b34d923737fb6e49" candidate_uuid: "d09ce4a3364c4498b724195111b25e5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74177132341f434ead6ad396bd220764" is_pre_election: true
I20240503 08:11:27.099241 2654 raft_consensus.cc:2454] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09ce4a3364c4498b724195111b25e5c in term 0.
I20240503 08:11:27.099829 2729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85a6284cfa642e3b34d923737fb6e49" candidate_uuid: "d09ce4a3364c4498b724195111b25e5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4631d7d664da4054baca9a54fde1c25c"
I20240503 08:11:27.099829 2654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85a6284cfa642e3b34d923737fb6e49" candidate_uuid: "d09ce4a3364c4498b724195111b25e5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74177132341f434ead6ad396bd220764"
I20240503 08:11:27.099998 2729 raft_consensus.cc:3037] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.100039 2654 raft_consensus.cc:3037] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:27.102034 2654 raft_consensus.cc:2454] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09ce4a3364c4498b724195111b25e5c in term 1.
I20240503 08:11:27.102252 2729 raft_consensus.cc:2454] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d09ce4a3364c4498b724195111b25e5c in term 1.
I20240503 08:11:27.102383 2768 leader_election.cc:302] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74177132341f434ead6ad396bd220764, d09ce4a3364c4498b724195111b25e5c; no voters:
I20240503 08:11:27.102697 2851 raft_consensus.cc:2781] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:27.103185 2851 raft_consensus.cc:686] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 1 LEADER]: Becoming Leader. State: Replica: d09ce4a3364c4498b724195111b25e5c, State: Running, Role: LEADER
I20240503 08:11:27.103499 2851 consensus_queue.cc:237] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } }
I20240503 08:11:27.105572 2552 catalog_manager.cc:5484] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c reported cstate change: term changed from 0 to 1, leader changed from <none> to d09ce4a3364c4498b724195111b25e5c (127.0.230.3). New cstate: current_term: 1 leader_uuid: "d09ce4a3364c4498b724195111b25e5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:27.492918 2851 consensus_queue.cc:1035] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:27.497738 2867 consensus_queue.cc:1035] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:27.592198 2857 consensus_queue.cc:1035] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:27.599138 2876 consensus_queue.cc:1035] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:27.695132 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:27.695758 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:27.695909 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:29.696300 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:29.696940 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:29.697068 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:31.697523 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:31.698163 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:31.698313 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:33.698701 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:33.699370 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:33.699529 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:35.699923 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:35.700523 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:35.700692 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:37.701097 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:37.701748 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:37.701879 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:38.171324 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:38.174212 2885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:38.174870 2886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:38.175657 2888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:38.175966 920 server_base.cc:1034] running on GCE node
I20240503 08:11:38.176223 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:38.176301 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:38.176358 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723898176337 us; error 0 us; skew 500 ppm
I20240503 08:11:38.176512 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:38.181159 920 webserver.cc:460] Webserver started at http://127.0.230.4:38489/ using document root <none> and password file <none>
I20240503 08:11:38.181391 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:38.181468 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:38.181610 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:38.182271 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-3-root/instance:
uuid: "05a47dba7ed6464ba80fea84ae7bf35c"
format_stamp: "Formatted at 2024-05-03 08:11:38 on dist-test-slave-2jf9"
I20240503 08:11:38.184720 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:38.186187 2893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.186625 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:38.186749 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-3-root
uuid: "05a47dba7ed6464ba80fea84ae7bf35c"
format_stamp: "Formatted at 2024-05-03 08:11:38 on dist-test-slave-2jf9"
I20240503 08:11:38.186867 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:38.201767 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:38.202330 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:38.202879 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:38.203764 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:38.203855 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.203944 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:38.204010 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.218396 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.4:39493
I20240503 08:11:38.218437 2956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.4:39493 every 8 connection(s)
I20240503 08:11:38.224854 2957 heartbeater.cc:347] Connected to a master server at 127.0.230.62:45567
I20240503 08:11:38.224977 2957 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:38.225298 2957 heartbeater.cc:511] Master 127.0.230.62:45567 requested a full tablet report, sending...
I20240503 08:11:38.226063 2552 ts_manager.cc:194] Registered new tserver with Master: 05a47dba7ed6464ba80fea84ae7bf35c (127.0.230.4:39493)
I20240503 08:11:38.226758 2552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57888
I20240503 08:11:38.716141 2803 consensus_queue.cc:237] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: true } }
I20240503 08:11:38.718485 2729 raft_consensus.cc:1261] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 1 FOLLOWER]: Refusing update from remote peer d09ce4a3364c4498b724195111b25e5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:38.718895 2654 raft_consensus.cc:1261] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [term 1 FOLLOWER]: Refusing update from remote peer d09ce4a3364c4498b724195111b25e5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:38.719275 2965 consensus_queue.cc:1035] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:38.719698 2964 consensus_queue.cc:1035] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:38.724936 2964 raft_consensus.cc:2932] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05a47dba7ed6464ba80fea84ae7bf35c (127.0.230.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: true } } }
W20240503 08:11:38.725761 2768 consensus_peers.cc:485] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c -> Peer 05a47dba7ed6464ba80fea84ae7bf35c (127.0.230.4:39493): Couldn't send request to peer 05a47dba7ed6464ba80fea84ae7bf35c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e85a6284cfa642e3b34d923737fb6e49. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:38.725751 2653 raft_consensus.cc:2932] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05a47dba7ed6464ba80fea84ae7bf35c (127.0.230.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: true } } }
I20240503 08:11:38.726434 2729 raft_consensus.cc:2932] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05a47dba7ed6464ba80fea84ae7bf35c (127.0.230.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: true } } }
I20240503 08:11:38.729818 2552 catalog_manager.cc:5484] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 reported cstate change: config changed from index -1 to 2, NON_VOTER 05a47dba7ed6464ba80fea84ae7bf35c (127.0.230.4) added. New cstate: current_term: 1 leader_uuid: "d09ce4a3364c4498b724195111b25e5c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: true } } }
I20240503 08:11:39.116616 2975 ts_tablet_manager.cc:902] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c: Initiating tablet copy from peer d09ce4a3364c4498b724195111b25e5c (127.0.230.3:42319)
I20240503 08:11:39.118510 2975 tablet_copy_client.cc:308] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.3:42319
I20240503 08:11:39.125859 2813 tablet_copy_service.cc:140] P d09ce4a3364c4498b724195111b25e5c: Received BeginTabletCopySession request for tablet e85a6284cfa642e3b34d923737fb6e49 from peer 05a47dba7ed6464ba80fea84ae7bf35c ({username='slave'} at 127.0.0.1:33718)
I20240503 08:11:39.126120 2813 tablet_copy_service.cc:161] P d09ce4a3364c4498b724195111b25e5c: Beginning new tablet copy session on tablet e85a6284cfa642e3b34d923737fb6e49 from peer 05a47dba7ed6464ba80fea84ae7bf35c at {username='slave'} at 127.0.0.1:33718: session id = 05a47dba7ed6464ba80fea84ae7bf35c-e85a6284cfa642e3b34d923737fb6e49
I20240503 08:11:39.128876 2813 tablet_copy_source_session.cc:205] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:39.130599 2975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85a6284cfa642e3b34d923737fb6e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:39.135426 2975 tablet_copy_client.cc:783] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:39.135731 2975 tablet_copy_client.cc:647] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:39.137893 2975 tablet_copy_client.cc:523] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:39.141039 2975 tablet_bootstrap.cc:492] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c: Bootstrap starting.
I20240503 08:11:39.147024 2975 tablet_bootstrap.cc:492] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:39.147385 2975 tablet_bootstrap.cc:492] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c: Bootstrap complete.
I20240503 08:11:39.147650 2975 ts_tablet_manager.cc:1367] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240503 08:11:39.148341 2975 raft_consensus.cc:348] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: true } }
I20240503 08:11:39.148591 2975 raft_consensus.cc:729] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05a47dba7ed6464ba80fea84ae7bf35c, State: Initialized, Role: LEARNER
I20240503 08:11:39.148831 2975 consensus_queue.cc:260] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: true } }
I20240503 08:11:39.149603 2975 ts_tablet_manager.cc:1398] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:39.149611 2957 heartbeater.cc:503] Master 127.0.230.62:45567 was elected leader, sending a full tablet report...
I20240503 08:11:39.150333 2813 tablet_copy_service.cc:342] P d09ce4a3364c4498b724195111b25e5c: Request end of tablet copy session 05a47dba7ed6464ba80fea84ae7bf35c-e85a6284cfa642e3b34d923737fb6e49 received from {username='slave'} at 127.0.0.1:33718
I20240503 08:11:39.150497 2813 tablet_copy_service.cc:433] P d09ce4a3364c4498b724195111b25e5c: ending tablet copy session 05a47dba7ed6464ba80fea84ae7bf35c-e85a6284cfa642e3b34d923737fb6e49 on tablet e85a6284cfa642e3b34d923737fb6e49 with peer 05a47dba7ed6464ba80fea84ae7bf35c
I20240503 08:11:39.545863 2932 raft_consensus.cc:1203] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:11:39.702193 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:39.702759 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:39.702922 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:40.130363 2984 raft_consensus.cc:1050] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c: attempting to promote NON_VOTER 05a47dba7ed6464ba80fea84ae7bf35c to VOTER
I20240503 08:11:40.131283 2984 consensus_queue.cc:237] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false } }
I20240503 08:11:40.133586 2932 raft_consensus.cc:1261] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c [term 1 LEARNER]: Refusing update from remote peer d09ce4a3364c4498b724195111b25e5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:11:40.134025 2653 raft_consensus.cc:1261] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [term 1 FOLLOWER]: Refusing update from remote peer d09ce4a3364c4498b724195111b25e5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:11:40.134582 2985 consensus_queue.cc:1035] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:40.135257 2985 consensus_queue.cc:1035] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:40.139627 2985 raft_consensus.cc:2932] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05a47dba7ed6464ba80fea84ae7bf35c (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false } } }
I20240503 08:11:40.140056 2653 raft_consensus.cc:2932] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05a47dba7ed6464ba80fea84ae7bf35c (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false } } }
I20240503 08:11:40.144130 2729 raft_consensus.cc:1261] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 1 FOLLOWER]: Refusing update from remote peer d09ce4a3364c4498b724195111b25e5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:11:40.144191 2550 catalog_manager.cc:5484] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 reported cstate change: config changed from index 2 to 3, 05a47dba7ed6464ba80fea84ae7bf35c (127.0.230.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d09ce4a3364c4498b724195111b25e5c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false } } }
I20240503 08:11:40.145018 2984 consensus_queue.cc:1035] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 08:11:40.146502 2932 raft_consensus.cc:2932] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05a47dba7ed6464ba80fea84ae7bf35c (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false } } }
I20240503 08:11:40.146793 2729 raft_consensus.cc:2932] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05a47dba7ed6464ba80fea84ae7bf35c (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "74177132341f434ead6ad396bd220764" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 46715 } attrs { replace: true } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false } } }
I20240503 08:11:40.724738 2803 consensus_queue.cc:237] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false } }
I20240503 08:11:40.726449 2932 raft_consensus.cc:1261] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c [term 1 FOLLOWER]: Refusing update from remote peer d09ce4a3364c4498b724195111b25e5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:40.726611 2729 raft_consensus.cc:1261] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 1 FOLLOWER]: Refusing update from remote peer d09ce4a3364c4498b724195111b25e5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:40.727084 2998 consensus_queue.cc:1035] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:40.727645 2998 consensus_queue.cc:1035] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:40.729635 2987 raft_consensus.cc:2932] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74177132341f434ead6ad396bd220764 (127.0.230.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false } } }
I20240503 08:11:40.730191 2932 raft_consensus.cc:2932] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74177132341f434ead6ad396bd220764 (127.0.230.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false } } }
I20240503 08:11:40.730325 2729 raft_consensus.cc:2932] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74177132341f434ead6ad396bd220764 (127.0.230.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false } } }
I20240503 08:11:40.735504 2538 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e85a6284cfa642e3b34d923737fb6e49 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:40.736166 2552 catalog_manager.cc:5484] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c reported cstate change: config changed from index 3 to 4, VOTER 74177132341f434ead6ad396bd220764 (127.0.230.1) evicted. New cstate: current_term: 1 leader_uuid: "d09ce4a3364c4498b724195111b25e5c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d09ce4a3364c4498b724195111b25e5c" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 42319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4631d7d664da4054baca9a54fde1c25c" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 41387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05a47dba7ed6464ba80fea84ae7bf35c" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 39493 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:40.747464 2644 tablet_service.cc:1514] Processing DeleteTablet for tablet e85a6284cfa642e3b34d923737fb6e49 with delete_type TABLET_DATA_TOMBSTONED (TS 74177132341f434ead6ad396bd220764 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48464
I20240503 08:11:40.748390 3006 tablet_replica.cc:330] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764: stopping tablet replica
I20240503 08:11:40.748615 3006 raft_consensus.cc:2229] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:40.748904 3006 raft_consensus.cc:2258] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:40.749714 3006 ts_tablet_manager.cc:1875] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:40.755117 3006 ts_tablet_manager.cc:1888] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:11:40.755230 3006 log.cc:1196] T e85a6284cfa642e3b34d923737fb6e49 P 74177132341f434ead6ad396bd220764: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723866726930-920-0/minicluster-data/ts-0-root/wals/e85a6284cfa642e3b34d923737fb6e49
I20240503 08:11:40.755847 2538 catalog_manager.cc:4830] TS 74177132341f434ead6ad396bd220764 (127.0.230.1:46715): tablet e85a6284cfa642e3b34d923737fb6e49 (table test-workload [id=994da41fea9743a7ad14e5185b4e61bb]) successfully deleted
I20240503 08:11:41.703374 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:41.704030 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:41.704135 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:43.704504 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:43.705123 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:43.705231 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:45.705550 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:45.706159 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:45.706261 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:47.706622 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:47.707257 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:47.707363 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:49.707755 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:49.708305 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:49.708400 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:51.708853 2605 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:11:51.709662 2605 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:11:51.709800 2605 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:11:52.308702 920 tablet_server.cc:178] TabletServer@127.0.230.1:0 shutting down...
I20240503 08:11:52.319145 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:52.319378 920 tablet_replica.cc:330] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764: stopping tablet replica
I20240503 08:11:52.319595 920 raft_consensus.cc:2229] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:52.319901 920 raft_consensus.cc:2258] T e70cca9326a246c3992b510dcce35050 P 74177132341f434ead6ad396bd220764 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:52.333799 920 tablet_server.cc:194] TabletServer@127.0.230.1:0 shutdown complete.
I20240503 08:11:52.336791 920 tablet_server.cc:178] TabletServer@127.0.230.2:0 shutting down...
I20240503 08:11:52.344303 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:52.344561 920 tablet_replica.cc:330] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c: stopping tablet replica
I20240503 08:11:52.344746 920 raft_consensus.cc:2229] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:52.344911 920 raft_consensus.cc:2258] T e70cca9326a246c3992b510dcce35050 P 4631d7d664da4054baca9a54fde1c25c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:52.345443 920 tablet_replica.cc:330] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c: stopping tablet replica
I20240503 08:11:52.345566 920 raft_consensus.cc:2229] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:52.345753 920 raft_consensus.cc:2258] T e85a6284cfa642e3b34d923737fb6e49 P 4631d7d664da4054baca9a54fde1c25c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:52.358927 920 tablet_server.cc:194] TabletServer@127.0.230.2:0 shutdown complete.
I20240503 08:11:52.361847 920 tablet_server.cc:178] TabletServer@127.0.230.3:0 shutting down...
I20240503 08:11:52.376266 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:52.376520 920 tablet_replica.cc:330] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c: stopping tablet replica
I20240503 08:11:52.376713 920 raft_consensus.cc:2229] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:52.376870 920 raft_consensus.cc:2258] T e70cca9326a246c3992b510dcce35050 P d09ce4a3364c4498b724195111b25e5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:52.377388 920 tablet_replica.cc:330] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c: stopping tablet replica
I20240503 08:11:52.377490 920 raft_consensus.cc:2229] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:52.377738 920 raft_consensus.cc:2258] T e85a6284cfa642e3b34d923737fb6e49 P d09ce4a3364c4498b724195111b25e5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:52.392830 920 tablet_server.cc:194] TabletServer@127.0.230.3:0 shutdown complete.
I20240503 08:11:52.396170 920 tablet_server.cc:178] TabletServer@127.0.230.4:0 shutting down...
I20240503 08:11:52.404850 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:52.405143 920 tablet_replica.cc:330] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c: stopping tablet replica
I20240503 08:11:52.405290 920 raft_consensus.cc:2229] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:52.405500 920 raft_consensus.cc:2258] T e85a6284cfa642e3b34d923737fb6e49 P 05a47dba7ed6464ba80fea84ae7bf35c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:52.418845 920 tablet_server.cc:194] TabletServer@127.0.230.4:0 shutdown complete.
I20240503 08:11:52.421934 920 master.cc:537] Master@127.0.230.62:45567 shutting down...
I20240503 08:11:52.430047 920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:52.430275 920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:52.430372 920 tablet_replica.cc:330] T 00000000000000000000000000000000 P d0ca07d9231a4c308d013354dab41b33: stopping tablet replica
I20240503 08:11:52.443280 920 master.cc:552] Master@127.0.230.62:45567 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26810 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240503 08:11:52.454553 920 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.230.62:40269
I20240503 08:11:52.455029 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:52.457628 3008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:52.458415 3009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:52.459031 920 server_base.cc:1034] running on GCE node
W20240503 08:11:52.459558 3011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:52.459832 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:52.459884 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:52.459913 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723912459910 us; error 0 us; skew 500 ppm
I20240503 08:11:52.460023 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:52.491237 920 webserver.cc:460] Webserver started at http://127.0.230.62:46159/ using document root <none> and password file <none>
I20240503 08:11:52.491531 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:52.491603 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:52.491719 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:52.492341 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723866726930-920-0/minicluster-data/master-0-root/instance:
uuid: "cf53a62ea1ad4b1685a67c6f2418229d"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-2jf9"
I20240503 08:11:52.494745 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 08:11:52.496342 3016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.496688 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:52.496783 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723866726930-920-0/minicluster-data/master-0-root
uuid: "cf53a62ea1ad4b1685a67c6f2418229d"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-2jf9"
I20240503 08:11:52.496891 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723866726930-920-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723866726930-920-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723866726930-920-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:52.516270 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:52.516990 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:52.530215 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.62:40269
I20240503 08:11:52.530249 3067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.62:40269 every 8 connection(s)
I20240503 08:11:52.531531 3068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:52.535346 3068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d: Bootstrap starting.
I20240503 08:11:52.536401 3068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:52.537724 3068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d: No bootstrap required, opened a new log
I20240503 08:11:52.538264 3068 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf53a62ea1ad4b1685a67c6f2418229d" member_type: VOTER }
I20240503 08:11:52.538388 3068 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:52.538426 3068 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf53a62ea1ad4b1685a67c6f2418229d, State: Initialized, Role: FOLLOWER
I20240503 08:11:52.538594 3068 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf53a62ea1ad4b1685a67c6f2418229d" member_type: VOTER }
I20240503 08:11:52.538707 3068 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:52.538751 3068 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:52.538800 3068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:52.540392 3068 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf53a62ea1ad4b1685a67c6f2418229d" member_type: VOTER }
I20240503 08:11:52.540550 3068 leader_election.cc:302] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf53a62ea1ad4b1685a67c6f2418229d; no voters:
I20240503 08:11:52.540866 3068 leader_election.cc:288] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:52.540972 3071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:52.541404 3071 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [term 1 LEADER]: Becoming Leader. State: Replica: cf53a62ea1ad4b1685a67c6f2418229d, State: Running, Role: LEADER
I20240503 08:11:52.541633 3068 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:52.541613 3071 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf53a62ea1ad4b1685a67c6f2418229d" member_type: VOTER }
I20240503 08:11:52.542385 3073 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf53a62ea1ad4b1685a67c6f2418229d. Latest consensus state: current_term: 1 leader_uuid: "cf53a62ea1ad4b1685a67c6f2418229d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf53a62ea1ad4b1685a67c6f2418229d" member_type: VOTER } }
I20240503 08:11:52.542594 3073 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:52.542704 3072 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf53a62ea1ad4b1685a67c6f2418229d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf53a62ea1ad4b1685a67c6f2418229d" member_type: VOTER } }
I20240503 08:11:52.542908 3072 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:52.543465 3076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:52.545154 3076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:52.546324 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:11:52.548182 3076 catalog_manager.cc:1347] Generated new cluster ID: 46f3216f8d7e41a0892879f46bc98fd3
I20240503 08:11:52.548254 3076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:52.555764 3076 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:52.556493 3076 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:52.561614 3076 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d: Generated new TSK 0
I20240503 08:11:52.561841 3076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:52.578537 920 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 9.4238e-05s
I20240503 08:12:04.659587 920 master.cc:537] Master@127.0.230.62:40269 shutting down...
I20240503 08:12:04.672200 920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:04.672466 920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:04.672573 920 tablet_replica.cc:330] T 00000000000000000000000000000000 P cf53a62ea1ad4b1685a67c6f2418229d: stopping tablet replica
I20240503 08:12:04.686276 920 master.cc:552] Master@127.0.230.62:40269 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12250 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240503 08:12:04.705430 920 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.230.62:43265
I20240503 08:12:04.705926 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:04.708617 3090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:04.709249 3091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:04.709640 3093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:04.710350 920 server_base.cc:1034] running on GCE node
I20240503 08:12:04.710610 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:04.710660 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:04.710685 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723924710681 us; error 0 us; skew 500 ppm
I20240503 08:12:04.710796 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:04.712015 920 webserver.cc:460] Webserver started at http://127.0.230.62:34275/ using document root <none> and password file <none>
I20240503 08:12:04.712203 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:04.712251 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:04.712347 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:04.712934 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/master-0-root/instance:
uuid: "bd89428e5e6f44248748ece37101abde"
format_stamp: "Formatted at 2024-05-03 08:12:04 on dist-test-slave-2jf9"
I20240503 08:12:04.715184 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:04.716708 3098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.717015 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:04.717114 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/master-0-root
uuid: "bd89428e5e6f44248748ece37101abde"
format_stamp: "Formatted at 2024-05-03 08:12:04 on dist-test-slave-2jf9"
I20240503 08:12:04.717221 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:04.741907 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:04.742444 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:04.752669 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.62:43265
I20240503 08:12:04.752689 3149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.62:43265 every 8 connection(s)
I20240503 08:12:04.753943 3150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:04.757860 3150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde: Bootstrap starting.
I20240503 08:12:04.759063 3150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:04.760622 3150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde: No bootstrap required, opened a new log
I20240503 08:12:04.761189 3150 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd89428e5e6f44248748ece37101abde" member_type: VOTER }
I20240503 08:12:04.761312 3150 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:04.761358 3150 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd89428e5e6f44248748ece37101abde, State: Initialized, Role: FOLLOWER
I20240503 08:12:04.761550 3150 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd89428e5e6f44248748ece37101abde" member_type: VOTER }
I20240503 08:12:04.761656 3150 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:12:04.761706 3150 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:12:04.761778 3150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:04.763562 3150 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd89428e5e6f44248748ece37101abde" member_type: VOTER }
I20240503 08:12:04.764039 3150 leader_election.cc:302] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd89428e5e6f44248748ece37101abde; no voters:
I20240503 08:12:04.764892 3150 leader_election.cc:288] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:12:04.765218 3153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:04.766216 3153 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [term 1 LEADER]: Becoming Leader. State: Replica: bd89428e5e6f44248748ece37101abde, State: Running, Role: LEADER
I20240503 08:12:04.766427 3153 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd89428e5e6f44248748ece37101abde" member_type: VOTER }
I20240503 08:12:04.766548 3150 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:12:04.767098 3154 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd89428e5e6f44248748ece37101abde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd89428e5e6f44248748ece37101abde" member_type: VOTER } }
I20240503 08:12:04.767164 3155 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd89428e5e6f44248748ece37101abde. Latest consensus state: current_term: 1 leader_uuid: "bd89428e5e6f44248748ece37101abde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd89428e5e6f44248748ece37101abde" member_type: VOTER } }
I20240503 08:12:04.767311 3154 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:04.767335 3155 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:04.771229 3158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:12:04.771504 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:12:04.772492 3158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:12:04.775094 3158 catalog_manager.cc:1347] Generated new cluster ID: 47762ecb9a0f417980823a3ef136a8b5
I20240503 08:12:04.775171 3158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:12:04.787649 3158 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:12:04.788414 3158 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:12:04.794808 3158 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde: Generated new TSK 0
I20240503 08:12:04.795047 3158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:12:04.804267 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:04.806958 3171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:04.807410 3172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:04.808044 3174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:04.808506 920 server_base.cc:1034] running on GCE node
I20240503 08:12:04.808717 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:04.808768 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:04.808795 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723924808792 us; error 0 us; skew 500 ppm
I20240503 08:12:04.808933 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:04.810082 920 webserver.cc:460] Webserver started at http://127.0.230.1:45805/ using document root <none> and password file <none>
I20240503 08:12:04.810282 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:04.810340 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:04.810467 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:04.811067 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-0-root/instance:
uuid: "88bf7a773ede40c8b1d6dfe393f7dab5"
format_stamp: "Formatted at 2024-05-03 08:12:04 on dist-test-slave-2jf9"
I20240503 08:12:04.813582 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:04.815052 3179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.815471 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:04.815574 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-0-root
uuid: "88bf7a773ede40c8b1d6dfe393f7dab5"
format_stamp: "Formatted at 2024-05-03 08:12:04 on dist-test-slave-2jf9"
I20240503 08:12:04.815687 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:04.825368 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:04.825816 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:04.826280 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:04.827036 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:04.827085 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.827164 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:04.827200 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.838225 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.1:35747
I20240503 08:12:04.838245 3241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.1:35747 every 8 connection(s)
I20240503 08:12:04.839702 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:04.842517 3246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:04.843466 3247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:04.843997 920 server_base.cc:1034] running on GCE node
W20240503 08:12:04.844473 3249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:04.844852 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:04.844923 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:04.844966 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723924844962 us; error 0 us; skew 500 ppm
I20240503 08:12:04.845109 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:04.846319 3242 heartbeater.cc:347] Connected to a master server at 127.0.230.62:43265
I20240503 08:12:04.846447 3242 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:04.846448 920 webserver.cc:460] Webserver started at http://127.0.230.2:46625/ using document root <none> and password file <none>
I20240503 08:12:04.846680 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:04.846757 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:04.846872 3242 heartbeater.cc:511] Master 127.0.230.62:43265 requested a full tablet report, sending...
I20240503 08:12:04.846885 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:04.847618 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-1-root/instance:
uuid: "1969f469dccf4acf9bff039f72496396"
format_stamp: "Formatted at 2024-05-03 08:12:04 on dist-test-slave-2jf9"
I20240503 08:12:04.847775 3115 ts_manager.cc:194] Registered new tserver with Master: 88bf7a773ede40c8b1d6dfe393f7dab5 (127.0.230.1:35747)
I20240503 08:12:04.848802 3115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58724
I20240503 08:12:04.850073 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:04.851423 3254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.851714 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 08:12:04.851823 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-1-root
uuid: "1969f469dccf4acf9bff039f72496396"
format_stamp: "Formatted at 2024-05-03 08:12:04 on dist-test-slave-2jf9"
I20240503 08:12:04.851919 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:04.864674 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:04.865135 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:04.865639 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:04.866387 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:04.866441 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.866511 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:04.866544 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.878829 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.2:39207
I20240503 08:12:04.878861 3316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.2:39207 every 8 connection(s)
I20240503 08:12:04.880261 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:04.882977 3320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:04.883492 3321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:04.884052 3323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:04.884207 920 server_base.cc:1034] running on GCE node
I20240503 08:12:04.884622 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:04.884703 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:04.884752 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723924884748 us; error 0 us; skew 500 ppm
I20240503 08:12:04.884896 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:04.886193 920 webserver.cc:460] Webserver started at http://127.0.230.3:37173/ using document root <none> and password file <none>
I20240503 08:12:04.886415 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:04.886472 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:04.886579 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:04.886787 3317 heartbeater.cc:347] Connected to a master server at 127.0.230.62:43265
I20240503 08:12:04.886914 3317 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:04.887306 3317 heartbeater.cc:511] Master 127.0.230.62:43265 requested a full tablet report, sending...
I20240503 08:12:04.887321 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-2-root/instance:
uuid: "98efd469fce34040b1abd0cbf43778bb"
format_stamp: "Formatted at 2024-05-03 08:12:04 on dist-test-slave-2jf9"
I20240503 08:12:04.888236 3115 ts_manager.cc:194] Registered new tserver with Master: 1969f469dccf4acf9bff039f72496396 (127.0.230.2:39207)
I20240503 08:12:04.889026 3115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58732
I20240503 08:12:04.890136 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:04.891654 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.891961 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:04.892057 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-2-root
uuid: "98efd469fce34040b1abd0cbf43778bb"
format_stamp: "Formatted at 2024-05-03 08:12:04 on dist-test-slave-2jf9"
I20240503 08:12:04.892158 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723866726930-920-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:04.904028 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:04.904600 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:04.905217 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:04.905987 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:04.906037 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.906100 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:04.906132 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:04.917830 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.3:36561
I20240503 08:12:04.917848 3390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.3:36561 every 8 connection(s)
I20240503 08:12:04.924253 3391 heartbeater.cc:347] Connected to a master server at 127.0.230.62:43265
I20240503 08:12:04.924379 3391 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:04.924768 3391 heartbeater.cc:511] Master 127.0.230.62:43265 requested a full tablet report, sending...
I20240503 08:12:04.925604 3115 ts_manager.cc:194] Registered new tserver with Master: 98efd469fce34040b1abd0cbf43778bb (127.0.230.3:36561)
I20240503 08:12:04.926306 3115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58742
I20240503 08:12:04.926446 920 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007826538s
I20240503 08:12:05.850127 3242 heartbeater.cc:503] Master 127.0.230.62:43265 was elected leader, sending a full tablet report...
I20240503 08:12:05.890446 3317 heartbeater.cc:503] Master 127.0.230.62:43265 was elected leader, sending a full tablet report...
I20240503 08:12:05.927554 3391 heartbeater.cc:503] Master 127.0.230.62:43265 was elected leader, sending a full tablet report...
I20240503 08:12:06.770892 3168 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:08.771252 3168 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:10.771739 3168 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:12.772207 3168 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:14.772634 3168 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:16.773078 3168 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:17.007833 920 tablet_server.cc:178] TabletServer@127.0.230.1:0 shutting down...
I20240503 08:12:17.019354 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:17.032966 920 tablet_server.cc:194] TabletServer@127.0.230.1:0 shutdown complete.
I20240503 08:12:17.035642 920 tablet_server.cc:178] TabletServer@127.0.230.2:0 shutting down...
I20240503 08:12:17.042924 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:17.055562 920 tablet_server.cc:194] TabletServer@127.0.230.2:0 shutdown complete.
I20240503 08:12:17.058274 920 tablet_server.cc:178] TabletServer@127.0.230.3:0 shutting down...
I20240503 08:12:17.065153 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:17.077651 920 tablet_server.cc:194] TabletServer@127.0.230.3:0 shutdown complete.
I20240503 08:12:17.080271 920 master.cc:537] Master@127.0.230.62:43265 shutting down...
I20240503 08:12:17.087103 920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:17.087347 920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:17.087491 920 tablet_replica.cc:330] T 00000000000000000000000000000000 P bd89428e5e6f44248748ece37101abde: stopping tablet replica
I20240503 08:12:17.100481 920 master.cc:552] Master@127.0.230.62:43265 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12413 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240503 08:12:17.119307 920 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.230.62:37045
I20240503 08:12:17.119835 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:17.122350 3395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:17.122943 3396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:17.123442 3398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:17.123805 920 server_base.cc:1034] running on GCE node
I20240503 08:12:17.124002 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:17.124049 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:17.124078 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723937124074 us; error 0 us; skew 500 ppm
I20240503 08:12:17.124214 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:17.129352 920 webserver.cc:460] Webserver started at http://127.0.230.62:35819/ using document root <none> and password file <none>
I20240503 08:12:17.129577 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:17.129639 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:17.129756 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:17.130360 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/master-0-root/instance:
uuid: "54fe249cf0db40a099b2d5c01b542e9d"
format_stamp: "Formatted at 2024-05-03 08:12:17 on dist-test-slave-2jf9"
I20240503 08:12:17.132802 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:17.134279 3403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.134582 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:17.134670 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/master-0-root
uuid: "54fe249cf0db40a099b2d5c01b542e9d"
format_stamp: "Formatted at 2024-05-03 08:12:17 on dist-test-slave-2jf9"
I20240503 08:12:17.134773 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:17.144937 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:17.145378 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:17.154649 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.62:37045
I20240503 08:12:17.154662 3454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.62:37045 every 8 connection(s)
I20240503 08:12:17.155854 3455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:17.159626 3455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d: Bootstrap starting.
I20240503 08:12:17.160604 3455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:17.161882 3455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d: No bootstrap required, opened a new log
I20240503 08:12:17.162369 3455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fe249cf0db40a099b2d5c01b542e9d" member_type: VOTER }
I20240503 08:12:17.162483 3455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:17.162528 3455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54fe249cf0db40a099b2d5c01b542e9d, State: Initialized, Role: FOLLOWER
I20240503 08:12:17.162671 3455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fe249cf0db40a099b2d5c01b542e9d" member_type: VOTER }
I20240503 08:12:17.162760 3455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:12:17.162796 3455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:12:17.162840 3455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:17.164453 3455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fe249cf0db40a099b2d5c01b542e9d" member_type: VOTER }
I20240503 08:12:17.164619 3455 leader_election.cc:302] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54fe249cf0db40a099b2d5c01b542e9d; no voters:
I20240503 08:12:17.164913 3455 leader_election.cc:288] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:12:17.165019 3458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:17.165375 3458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [term 1 LEADER]: Becoming Leader. State: Replica: 54fe249cf0db40a099b2d5c01b542e9d, State: Running, Role: LEADER
I20240503 08:12:17.165612 3455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:12:17.165570 3458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fe249cf0db40a099b2d5c01b542e9d" member_type: VOTER }
I20240503 08:12:17.166318 3460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54fe249cf0db40a099b2d5c01b542e9d. Latest consensus state: current_term: 1 leader_uuid: "54fe249cf0db40a099b2d5c01b542e9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fe249cf0db40a099b2d5c01b542e9d" member_type: VOTER } }
I20240503 08:12:17.166352 3459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54fe249cf0db40a099b2d5c01b542e9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fe249cf0db40a099b2d5c01b542e9d" member_type: VOTER } }
I20240503 08:12:17.166451 3460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:17.166486 3459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:17.178066 3463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:12:17.179875 3463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:12:17.180259 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:12:17.182627 3463 catalog_manager.cc:1347] Generated new cluster ID: a76de1798fff499b96f453063cf91f8e
I20240503 08:12:17.182695 3463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:12:17.195534 3463 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:12:17.196317 3463 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:12:17.205106 3463 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d: Generated new TSK 0
I20240503 08:12:17.205339 3463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:12:17.212873 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:17.215883 3476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:17.216238 3477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:17.216670 3479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:17.217392 920 server_base.cc:1034] running on GCE node
I20240503 08:12:17.217626 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:17.217680 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:17.217715 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723937217710 us; error 0 us; skew 500 ppm
I20240503 08:12:17.217849 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:17.219018 920 webserver.cc:460] Webserver started at http://127.0.230.1:41595/ using document root <none> and password file <none>
I20240503 08:12:17.219213 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:17.219272 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:17.219384 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:17.220052 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-0-root/instance:
uuid: "ec34567fd105432fbea27d8a5d46df70"
format_stamp: "Formatted at 2024-05-03 08:12:17 on dist-test-slave-2jf9"
I20240503 08:12:17.222404 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:17.223976 3484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.224292 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:17.224380 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-0-root
uuid: "ec34567fd105432fbea27d8a5d46df70"
format_stamp: "Formatted at 2024-05-03 08:12:17 on dist-test-slave-2jf9"
I20240503 08:12:17.224489 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:17.234961 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:17.235493 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:17.236008 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:17.236835 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:17.236892 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.236958 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:17.236989 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.247236 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.1:36823
I20240503 08:12:17.247260 3546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.1:36823 every 8 connection(s)
I20240503 08:12:17.248730 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:17.251724 3551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:17.252194 3552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:17.252647 3554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:17.253042 920 server_base.cc:1034] running on GCE node
I20240503 08:12:17.253262 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:17.253311 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:17.253341 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723937253337 us; error 0 us; skew 500 ppm
I20240503 08:12:17.253486 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:17.254899 920 webserver.cc:460] Webserver started at http://127.0.230.2:45129/ using document root <none> and password file <none>
I20240503 08:12:17.255110 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:17.255172 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:17.255285 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:17.255359 3547 heartbeater.cc:347] Connected to a master server at 127.0.230.62:37045
I20240503 08:12:17.255504 3547 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:17.255856 3547 heartbeater.cc:511] Master 127.0.230.62:37045 requested a full tablet report, sending...
I20240503 08:12:17.255968 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-1-root/instance:
uuid: "265fd638407646b0a01565e39c506d64"
format_stamp: "Formatted at 2024-05-03 08:12:17 on dist-test-slave-2jf9"
I20240503 08:12:17.256657 3420 ts_manager.cc:194] Registered new tserver with Master: ec34567fd105432fbea27d8a5d46df70 (127.0.230.1:36823)
I20240503 08:12:17.257642 3420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42448
I20240503 08:12:17.258404 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 08:12:17.259953 3559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.260254 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:12:17.260344 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-1-root
uuid: "265fd638407646b0a01565e39c506d64"
format_stamp: "Formatted at 2024-05-03 08:12:17 on dist-test-slave-2jf9"
I20240503 08:12:17.260440 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:17.275182 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:17.275753 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:17.276252 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:17.277026 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:17.277081 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.277145 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:17.277177 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.288107 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.2:34017
I20240503 08:12:17.288125 3621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.2:34017 every 8 connection(s)
I20240503 08:12:17.289569 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:17.292543 3625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:17.293036 3626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:17.293550 920 server_base.cc:1034] running on GCE node
W20240503 08:12:17.293922 3628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:17.294327 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:17.294400 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:17.294433 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723937294430 us; error 0 us; skew 500 ppm
I20240503 08:12:17.294574 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:17.295437 3622 heartbeater.cc:347] Connected to a master server at 127.0.230.62:37045
I20240503 08:12:17.295557 3622 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:17.295902 920 webserver.cc:460] Webserver started at http://127.0.230.3:32773/ using document root <none> and password file <none>
I20240503 08:12:17.295907 3622 heartbeater.cc:511] Master 127.0.230.62:37045 requested a full tablet report, sending...
I20240503 08:12:17.296115 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:17.296171 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:17.296325 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:17.296720 3420 ts_manager.cc:194] Registered new tserver with Master: 265fd638407646b0a01565e39c506d64 (127.0.230.2:34017)
I20240503 08:12:17.296977 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-2-root/instance:
uuid: "ea7ab580c6254693b29ebc26afda45f3"
format_stamp: "Formatted at 2024-05-03 08:12:17 on dist-test-slave-2jf9"
I20240503 08:12:17.297435 3420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42462
I20240503 08:12:17.299310 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:17.300788 3633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.301067 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:17.301168 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-2-root
uuid: "ea7ab580c6254693b29ebc26afda45f3"
format_stamp: "Formatted at 2024-05-03 08:12:17 on dist-test-slave-2jf9"
I20240503 08:12:17.301256 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:17.323822 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:17.324366 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:17.324883 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:17.325711 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:17.325765 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.325834 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:17.325865 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.336876 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.3:41153
I20240503 08:12:17.336892 3695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.3:41153 every 8 connection(s)
I20240503 08:12:17.338301 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:17.341995 3700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:17.342101 3699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:17.342489 3702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:17.342888 920 server_base.cc:1034] running on GCE node
I20240503 08:12:17.343178 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:17.343235 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:17.343267 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723937343264 us; error 0 us; skew 500 ppm
I20240503 08:12:17.343429 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:17.345328 920 webserver.cc:460] Webserver started at http://127.0.230.4:46553/ using document root <none> and password file <none>
I20240503 08:12:17.345587 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:17.345669 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:17.345788 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:17.346457 3696 heartbeater.cc:347] Connected to a master server at 127.0.230.62:37045
I20240503 08:12:17.346479 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-3-root/instance:
uuid: "511a8dac9d1f4650ae3c30dd3b225468"
format_stamp: "Formatted at 2024-05-03 08:12:17 on dist-test-slave-2jf9"
I20240503 08:12:17.346554 3696 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:17.346853 3696 heartbeater.cc:511] Master 127.0.230.62:37045 requested a full tablet report, sending...
I20240503 08:12:17.347682 3420 ts_manager.cc:194] Registered new tserver with Master: ea7ab580c6254693b29ebc26afda45f3 (127.0.230.3:41153)
I20240503 08:12:17.348376 3420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42464
I20240503 08:12:17.349074 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:17.350625 3707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.350946 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:12:17.351032 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-3-root
uuid: "511a8dac9d1f4650ae3c30dd3b225468"
format_stamp: "Formatted at 2024-05-03 08:12:17 on dist-test-slave-2jf9"
I20240503 08:12:17.351138 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723866726930-920-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:17.392593 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:17.393163 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:17.393684 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:17.394449 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:17.394505 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.394579 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:17.394614 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:17.405372 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.4:41031
I20240503 08:12:17.405413 3769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.4:41031 every 8 connection(s)
I20240503 08:12:17.411682 3770 heartbeater.cc:347] Connected to a master server at 127.0.230.62:37045
I20240503 08:12:17.411814 3770 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:17.412120 3770 heartbeater.cc:511] Master 127.0.230.62:37045 requested a full tablet report, sending...
I20240503 08:12:17.412915 3420 ts_manager.cc:194] Registered new tserver with Master: 511a8dac9d1f4650ae3c30dd3b225468 (127.0.230.4:41031)
I20240503 08:12:17.412940 920 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.00678962s
I20240503 08:12:17.413666 3420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42476
I20240503 08:12:18.258998 3547 heartbeater.cc:503] Master 127.0.230.62:37045 was elected leader, sending a full tablet report...
I20240503 08:12:18.298575 3622 heartbeater.cc:503] Master 127.0.230.62:37045 was elected leader, sending a full tablet report...
I20240503 08:12:18.349694 3696 heartbeater.cc:503] Master 127.0.230.62:37045 was elected leader, sending a full tablet report...
I20240503 08:12:18.414862 3770 heartbeater.cc:503] Master 127.0.230.62:37045 was elected leader, sending a full tablet report...
I20240503 08:12:18.446921 920 test_util.cc:268] Using random seed: 1195191876
I20240503 08:12:18.457448 3420 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42496:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 08:12:18.480113 3512 tablet_service.cc:1467] Processing CreateTablet for tablet fb1d73d4845945229b12ac3033a71bf6 (DEFAULT_TABLE table=test-workload [id=209a3dc8d5d046989a424580265e0be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.480656 3512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1d73d4845945229b12ac3033a71bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.480899 3661 tablet_service.cc:1467] Processing CreateTablet for tablet fb1d73d4845945229b12ac3033a71bf6 (DEFAULT_TABLE table=test-workload [id=209a3dc8d5d046989a424580265e0be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.481261 3661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1d73d4845945229b12ac3033a71bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.486351 3735 tablet_service.cc:1467] Processing CreateTablet for tablet fb1d73d4845945229b12ac3033a71bf6 (DEFAULT_TABLE table=test-workload [id=209a3dc8d5d046989a424580265e0be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.486769 3735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1d73d4845945229b12ac3033a71bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.487319 3791 tablet_bootstrap.cc:492] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3: Bootstrap starting.
I20240503 08:12:18.488735 3791 tablet_bootstrap.cc:654] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.489214 3792 tablet_bootstrap.cc:492] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70: Bootstrap starting.
I20240503 08:12:18.490995 3792 tablet_bootstrap.cc:654] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.492519 3791 tablet_bootstrap.cc:492] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3: No bootstrap required, opened a new log
I20240503 08:12:18.492552 3795 tablet_bootstrap.cc:492] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468: Bootstrap starting.
I20240503 08:12:18.492729 3791 ts_tablet_manager.cc:1367] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:12:18.493455 3791 raft_consensus.cc:348] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea7ab580c6254693b29ebc26afda45f3" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 41153 } } peers { permanent_uuid: "ec34567fd105432fbea27d8a5d46df70" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 36823 } } peers { permanent_uuid: "511a8dac9d1f4650ae3c30dd3b225468" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 41031 } }
I20240503 08:12:18.493667 3791 raft_consensus.cc:374] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.493738 3792 tablet_bootstrap.cc:492] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70: No bootstrap required, opened a new log
I20240503 08:12:18.493758 3791 raft_consensus.cc:729] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea7ab580c6254693b29ebc26afda45f3, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.493862 3792 ts_tablet_manager.cc:1367] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 08:12:18.494036 3791 consensus_queue.cc:260] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea7ab580c6254693b29ebc26afda45f3" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 41153 } } peers { permanent_uuid: "ec34567fd105432fbea27d8a5d46df70" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 36823 } } peers { permanent_uuid: "511a8dac9d1f4650ae3c30dd3b225468" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 41031 } }
I20240503 08:12:18.494225 3795 tablet_bootstrap.cc:654] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.494576 3792 raft_consensus.cc:348] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea7ab580c6254693b29ebc26afda45f3" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 41153 } } peers { permanent_uuid: "ec34567fd105432fbea27d8a5d46df70" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 36823 } } peers { permanent_uuid: "511a8dac9d1f4650ae3c30dd3b225468" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 41031 } }
I20240503 08:12:18.494786 3791 ts_tablet_manager.cc:1398] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:18.494822 3792 raft_consensus.cc:374] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.494932 3792 raft_consensus.cc:729] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec34567fd105432fbea27d8a5d46df70, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.495126 3792 consensus_queue.cc:260] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea7ab580c6254693b29ebc26afda45f3" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 41153 } } peers { permanent_uuid: "ec34567fd105432fbea27d8a5d46df70" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 36823 } } peers { permanent_uuid: "511a8dac9d1f4650ae3c30dd3b225468" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 41031 } }
I20240503 08:12:18.495877 3792 ts_tablet_manager.cc:1398] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:18.498322 3795 tablet_bootstrap.cc:492] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468: No bootstrap required, opened a new log
I20240503 08:12:18.498507 3795 ts_tablet_manager.cc:1367] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:18.499074 3795 raft_consensus.cc:348] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea7ab580c6254693b29ebc26afda45f3" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 41153 } } peers { permanent_uuid: "ec34567fd105432fbea27d8a5d46df70" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 36823 } } peers { permanent_uuid: "511a8dac9d1f4650ae3c30dd3b225468" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 41031 } }
W20240503 08:12:18.499203 3548 tablet.cc:2369] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:18.499253 3795 raft_consensus.cc:374] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.499331 3795 raft_consensus.cc:729] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 511a8dac9d1f4650ae3c30dd3b225468, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.499567 3795 consensus_queue.cc:260] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea7ab580c6254693b29ebc26afda45f3" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 41153 } } peers { permanent_uuid: "ec34567fd105432fbea27d8a5d46df70" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 36823 } } peers { permanent_uuid: "511a8dac9d1f4650ae3c30dd3b225468" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 41031 } }
I20240503 08:12:18.500164 3795 ts_tablet_manager.cc:1398] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:18.665922 3798 raft_consensus.cc:482] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:18.666144 3798 raft_consensus.cc:504] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea7ab580c6254693b29ebc26afda45f3" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 41153 } } peers { permanent_uuid: "ec34567fd105432fbea27d8a5d46df70" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 36823 } } peers { permanent_uuid: "511a8dac9d1f4650ae3c30dd3b225468" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 41031 } }
I20240503 08:12:18.666965 3798 leader_election.cc:288] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea7ab580c6254693b29ebc26afda45f3 (127.0.230.3:41153), 511a8dac9d1f4650ae3c30dd3b225468 (127.0.230.4:41031)
I20240503 08:12:18.673074 3671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1d73d4845945229b12ac3033a71bf6" candidate_uuid: "ec34567fd105432fbea27d8a5d46df70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea7ab580c6254693b29ebc26afda45f3" is_pre_election: true
I20240503 08:12:18.673074 3745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1d73d4845945229b12ac3033a71bf6" candidate_uuid: "ec34567fd105432fbea27d8a5d46df70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "511a8dac9d1f4650ae3c30dd3b225468" is_pre_election: true
I20240503 08:12:18.673408 3745 raft_consensus.cc:2454] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec34567fd105432fbea27d8a5d46df70 in term 0.
I20240503 08:12:18.673406 3671 raft_consensus.cc:2454] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec34567fd105432fbea27d8a5d46df70 in term 0.
I20240503 08:12:18.673785 3486 leader_election.cc:302] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 511a8dac9d1f4650ae3c30dd3b225468, ec34567fd105432fbea27d8a5d46df70; no voters:
I20240503 08:12:18.674026 3798 raft_consensus.cc:2781] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:18.674129 3798 raft_consensus.cc:482] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.674211 3798 raft_consensus.cc:3037] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.676127 3798 raft_consensus.cc:504] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea7ab580c6254693b29ebc26afda45f3" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 41153 } } peers { permanent_uuid: "ec34567fd105432fbea27d8a5d46df70" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 36823 } } peers { permanent_uuid: "511a8dac9d1f4650ae3c30dd3b225468" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 41031 } }
I20240503 08:12:18.676709 3798 leader_election.cc:288] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [CANDIDATE]: Term 1 election: Requested vote from peers ea7ab580c6254693b29ebc26afda45f3 (127.0.230.3:41153), 511a8dac9d1f4650ae3c30dd3b225468 (127.0.230.4:41031)
I20240503 08:12:18.677093 3671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1d73d4845945229b12ac3033a71bf6" candidate_uuid: "ec34567fd105432fbea27d8a5d46df70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea7ab580c6254693b29ebc26afda45f3"
I20240503 08:12:18.677161 3745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1d73d4845945229b12ac3033a71bf6" candidate_uuid: "ec34567fd105432fbea27d8a5d46df70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "511a8dac9d1f4650ae3c30dd3b225468"
I20240503 08:12:18.677260 3671 raft_consensus.cc:3037] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.677342 3745 raft_consensus.cc:3037] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.679098 3745 raft_consensus.cc:2454] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec34567fd105432fbea27d8a5d46df70 in term 1.
I20240503 08:12:18.679148 3671 raft_consensus.cc:2454] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec34567fd105432fbea27d8a5d46df70 in term 1.
I20240503 08:12:18.679442 3486 leader_election.cc:302] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 511a8dac9d1f4650ae3c30dd3b225468, ec34567fd105432fbea27d8a5d46df70; no voters:
I20240503 08:12:18.679689 3798 raft_consensus.cc:2781] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:18.680101 3798 raft_consensus.cc:686] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 1 LEADER]: Becoming Leader. State: Replica: ec34567fd105432fbea27d8a5d46df70, State: Running, Role: LEADER
I20240503 08:12:18.680342 3798 consensus_queue.cc:237] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea7ab580c6254693b29ebc26afda45f3" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 41153 } } peers { permanent_uuid: "ec34567fd105432fbea27d8a5d46df70" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 36823 } } peers { permanent_uuid: "511a8dac9d1f4650ae3c30dd3b225468" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 41031 } }
I20240503 08:12:18.682726 3420 catalog_manager.cc:5484] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec34567fd105432fbea27d8a5d46df70 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "ec34567fd105432fbea27d8a5d46df70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea7ab580c6254693b29ebc26afda45f3" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 41153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec34567fd105432fbea27d8a5d46df70" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 36823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "511a8dac9d1f4650ae3c30dd3b225468" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 41031 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:19.112886 3798 consensus_queue.cc:1035] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea7ab580c6254693b29ebc26afda45f3" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 41153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:19.115335 3798 consensus_queue.cc:1035] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "511a8dac9d1f4650ae3c30dd3b225468" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 41031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:19.180100 3473 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:19.180508 3473 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:19.180622 3473 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:21.181033 3473 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:21.181489 3473 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:21.181623 3473 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:23.182029 3473 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:23.182477 3473 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:23.182605 3473 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:25.183001 3473 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:25.183471 3473 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:25.183594 3473 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:27.183993 3473 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:27.184448 3473 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:27.184576 3473 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:29.184973 3473 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:29.185431 3473 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:29.185559 3473 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:29.754120 920 tablet_server.cc:178] TabletServer@127.0.230.1:0 shutting down...
I20240503 08:12:29.763012 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:29.763309 920 tablet_replica.cc:330] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70: stopping tablet replica
I20240503 08:12:29.763543 920 raft_consensus.cc:2229] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:29.763852 920 raft_consensus.cc:2258] T fb1d73d4845945229b12ac3033a71bf6 P ec34567fd105432fbea27d8a5d46df70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:29.777395 920 tablet_server.cc:194] TabletServer@127.0.230.1:0 shutdown complete.
I20240503 08:12:29.780333 920 tablet_server.cc:178] TabletServer@127.0.230.2:0 shutting down...
I20240503 08:12:29.796247 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:29.808712 920 tablet_server.cc:194] TabletServer@127.0.230.2:0 shutdown complete.
I20240503 08:12:29.811188 920 tablet_server.cc:178] TabletServer@127.0.230.3:0 shutting down...
I20240503 08:12:29.818261 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:29.818523 920 tablet_replica.cc:330] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3: stopping tablet replica
I20240503 08:12:29.818682 920 raft_consensus.cc:2229] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:29.818854 920 raft_consensus.cc:2258] T fb1d73d4845945229b12ac3033a71bf6 P ea7ab580c6254693b29ebc26afda45f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:29.832010 920 tablet_server.cc:194] TabletServer@127.0.230.3:0 shutdown complete.
I20240503 08:12:29.834709 920 tablet_server.cc:178] TabletServer@127.0.230.4:0 shutting down...
I20240503 08:12:29.842464 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:29.842697 920 tablet_replica.cc:330] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468: stopping tablet replica
I20240503 08:12:29.842857 920 raft_consensus.cc:2229] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:29.842994 920 raft_consensus.cc:2258] T fb1d73d4845945229b12ac3033a71bf6 P 511a8dac9d1f4650ae3c30dd3b225468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:29.855995 920 tablet_server.cc:194] TabletServer@127.0.230.4:0 shutdown complete.
I20240503 08:12:29.858927 920 master.cc:537] Master@127.0.230.62:37045 shutting down...
I20240503 08:12:29.865485 920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:29.865705 920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:29.865818 920 tablet_replica.cc:330] T 00000000000000000000000000000000 P 54fe249cf0db40a099b2d5c01b542e9d: stopping tablet replica
I20240503 08:12:29.878700 920 master.cc:552] Master@127.0.230.62:37045 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12769 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240503 08:12:29.889345 920 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.230.62:40169
I20240503 08:12:29.889830 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:29.892225 3814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:29.892732 3815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:29.893260 3817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:29.893747 920 server_base.cc:1034] running on GCE node
I20240503 08:12:29.893935 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:29.893981 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:29.894009 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723949894006 us; error 0 us; skew 500 ppm
I20240503 08:12:29.894140 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:29.898926 920 webserver.cc:460] Webserver started at http://127.0.230.62:34551/ using document root <none> and password file <none>
I20240503 08:12:29.899139 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:29.899199 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:29.899313 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:29.899966 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/master-0-root/instance:
uuid: "ee05e48b71004d8bbb955f26ac0b7b56"
format_stamp: "Formatted at 2024-05-03 08:12:29 on dist-test-slave-2jf9"
I20240503 08:12:29.902271 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:29.903780 3822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:29.904083 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:29.904174 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/master-0-root
uuid: "ee05e48b71004d8bbb955f26ac0b7b56"
format_stamp: "Formatted at 2024-05-03 08:12:29 on dist-test-slave-2jf9"
I20240503 08:12:29.904279 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:29.921463 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:29.921977 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:29.931548 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.62:40169
I20240503 08:12:29.931558 3873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.62:40169 every 8 connection(s)
I20240503 08:12:29.932785 3874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:29.936712 3874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56: Bootstrap starting.
I20240503 08:12:29.937855 3874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:29.939180 3874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56: No bootstrap required, opened a new log
I20240503 08:12:29.939774 3874 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee05e48b71004d8bbb955f26ac0b7b56" member_type: VOTER }
I20240503 08:12:29.939889 3874 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:29.939937 3874 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee05e48b71004d8bbb955f26ac0b7b56, State: Initialized, Role: FOLLOWER
I20240503 08:12:29.940111 3874 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee05e48b71004d8bbb955f26ac0b7b56" member_type: VOTER }
I20240503 08:12:29.940214 3874 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:12:29.940263 3874 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:12:29.940327 3874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:29.941934 3874 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee05e48b71004d8bbb955f26ac0b7b56" member_type: VOTER }
I20240503 08:12:29.942104 3874 leader_election.cc:302] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee05e48b71004d8bbb955f26ac0b7b56; no voters:
I20240503 08:12:29.942441 3874 leader_election.cc:288] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:12:29.942523 3877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:29.942936 3877 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [term 1 LEADER]: Becoming Leader. State: Replica: ee05e48b71004d8bbb955f26ac0b7b56, State: Running, Role: LEADER
I20240503 08:12:29.943138 3877 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee05e48b71004d8bbb955f26ac0b7b56" member_type: VOTER }
I20240503 08:12:29.943262 3874 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:12:29.943842 3878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee05e48b71004d8bbb955f26ac0b7b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee05e48b71004d8bbb955f26ac0b7b56" member_type: VOTER } }
I20240503 08:12:29.943914 3879 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee05e48b71004d8bbb955f26ac0b7b56. Latest consensus state: current_term: 1 leader_uuid: "ee05e48b71004d8bbb955f26ac0b7b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee05e48b71004d8bbb955f26ac0b7b56" member_type: VOTER } }
I20240503 08:12:29.944083 3878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:29.944106 3879 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:29.945401 3882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:12:29.947085 3882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:12:29.947599 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:12:29.949703 3882 catalog_manager.cc:1347] Generated new cluster ID: 122b160248f4452cb7d86710797593de
I20240503 08:12:29.949775 3882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:12:29.979161 3882 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:12:29.979972 3882 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:12:29.985545 3882 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56: Generated new TSK 0
I20240503 08:12:29.985774 3882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:12:30.012300 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:30.015045 3895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:30.015584 3896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:30.015976 3898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:30.016283 920 server_base.cc:1034] running on GCE node
I20240503 08:12:30.016472 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:30.016530 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:30.016571 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723950016567 us; error 0 us; skew 500 ppm
I20240503 08:12:30.016703 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:30.017856 920 webserver.cc:460] Webserver started at http://127.0.230.1:45687/ using document root <none> and password file <none>
I20240503 08:12:30.018064 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:30.018123 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:30.018236 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:30.018841 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-0-root/instance:
uuid: "cd35f7bbd58d4202b068e45a23696146"
format_stamp: "Formatted at 2024-05-03 08:12:30 on dist-test-slave-2jf9"
I20240503 08:12:30.021157 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:30.022605 3903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:30.022948 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:30.023028 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-0-root
uuid: "cd35f7bbd58d4202b068e45a23696146"
format_stamp: "Formatted at 2024-05-03 08:12:30 on dist-test-slave-2jf9"
I20240503 08:12:30.023123 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:30.045907 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:30.046404 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:30.046862 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:30.047725 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:30.047782 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:30.047853 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:30.047885 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:30.059082 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.1:35145
I20240503 08:12:30.059096 3965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.1:35145 every 8 connection(s)
I20240503 08:12:30.060580 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:30.063364 3970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:30.063899 3971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:30.064752 3973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:30.064961 920 server_base.cc:1034] running on GCE node
I20240503 08:12:30.065263 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:30.065354 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:30.065397 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723950065392 us; error 0 us; skew 500 ppm
I20240503 08:12:30.065567 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:30.066859 920 webserver.cc:460] Webserver started at http://127.0.230.2:41507/ using document root <none> and password file <none>
I20240503 08:12:30.067068 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:30.067137 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:30.067170 3966 heartbeater.cc:347] Connected to a master server at 127.0.230.62:40169
I20240503 08:12:30.067246 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:30.067277 3966 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:30.067683 3966 heartbeater.cc:511] Master 127.0.230.62:40169 requested a full tablet report, sending...
I20240503 08:12:30.067986 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-1-root/instance:
uuid: "982215fe60de4d1496b1919b1084c4c8"
format_stamp: "Formatted at 2024-05-03 08:12:30 on dist-test-slave-2jf9"
I20240503 08:12:30.068542 3839 ts_manager.cc:194] Registered new tserver with Master: cd35f7bbd58d4202b068e45a23696146 (127.0.230.1:35145)
I20240503 08:12:30.069911 3839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56192
I20240503 08:12:30.070847 920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:30.072381 3978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:30.072676 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:30.072775 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-1-root
uuid: "982215fe60de4d1496b1919b1084c4c8"
format_stamp: "Formatted at 2024-05-03 08:12:30 on dist-test-slave-2jf9"
I20240503 08:12:30.072867 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:30.105584 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:30.106099 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:30.106573 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:30.107385 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:30.107468 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:30.107540 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:30.107579 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:30.118094 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.2:37571
I20240503 08:12:30.118119 4040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.2:37571 every 8 connection(s)
I20240503 08:12:30.119516 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:30.122246 4044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:30.122687 4045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:30.123087 4047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:30.123438 920 server_base.cc:1034] running on GCE node
I20240503 08:12:30.125452 4041 heartbeater.cc:347] Connected to a master server at 127.0.230.62:40169
I20240503 08:12:30.125567 4041 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:30.125885 4041 heartbeater.cc:511] Master 127.0.230.62:40169 requested a full tablet report, sending...
I20240503 08:12:30.126657 3839 ts_manager.cc:194] Registered new tserver with Master: 982215fe60de4d1496b1919b1084c4c8 (127.0.230.2:37571)
I20240503 08:12:30.127369 3839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56194
I20240503 08:12:30.128206 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:30.128263 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:30.128291 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723950128287 us; error 0 us; skew 500 ppm
I20240503 08:12:30.128407 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:30.129493 920 webserver.cc:460] Webserver started at http://127.0.230.3:40497/ using document root <none> and password file <none>
I20240503 08:12:30.129678 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:30.129736 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:30.129848 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:30.130379 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-2-root/instance:
uuid: "ffb00f793a4d40b4bacff7f5457d1c30"
format_stamp: "Formatted at 2024-05-03 08:12:30 on dist-test-slave-2jf9"
I20240503 08:12:30.132493 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:30.133836 4052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:30.134152 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:30.134241 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-2-root
uuid: "ffb00f793a4d40b4bacff7f5457d1c30"
format_stamp: "Formatted at 2024-05-03 08:12:30 on dist-test-slave-2jf9"
I20240503 08:12:30.134325 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723866726930-920-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:30.141667 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:30.142066 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:30.142465 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:30.143128 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:30.143177 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:30.143234 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:30.143263 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:30.152743 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.3:44451
I20240503 08:12:30.152763 4114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.3:44451 every 8 connection(s)
I20240503 08:12:30.158185 4115 heartbeater.cc:347] Connected to a master server at 127.0.230.62:40169
I20240503 08:12:30.158293 4115 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:30.158571 4115 heartbeater.cc:511] Master 127.0.230.62:40169 requested a full tablet report, sending...
I20240503 08:12:30.159304 3839 ts_manager.cc:194] Registered new tserver with Master: ffb00f793a4d40b4bacff7f5457d1c30 (127.0.230.3:44451)
I20240503 08:12:30.160002 3839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56196
I20240503 08:12:30.160245 920 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006717795s
I20240503 08:12:31.071312 3966 heartbeater.cc:503] Master 127.0.230.62:40169 was elected leader, sending a full tablet report...
I20240503 08:12:31.128543 4041 heartbeater.cc:503] Master 127.0.230.62:40169 was elected leader, sending a full tablet report...
I20240503 08:12:31.161062 4115 heartbeater.cc:503] Master 127.0.230.62:40169 was elected leader, sending a full tablet report...
I20240503 08:12:31.192790 920 test_util.cc:268] Using random seed: 1207937743
I20240503 08:12:31.204463 3839 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56218:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:12:31.204970 3839 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:12:31.226205 3931 tablet_service.cc:1467] Processing CreateTablet for tablet f727ac7c101f4375bf0a447c842a4ce2 (DEFAULT_TABLE table=test-workload [id=d4e73938d0bd4894a1b19c09e62f6a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.226666 3931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f727ac7c101f4375bf0a447c842a4ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.226202 4080 tablet_service.cc:1467] Processing CreateTablet for tablet f727ac7c101f4375bf0a447c842a4ce2 (DEFAULT_TABLE table=test-workload [id=d4e73938d0bd4894a1b19c09e62f6a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.228256 4080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f727ac7c101f4375bf0a447c842a4ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.231808 4136 tablet_bootstrap.cc:492] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146: Bootstrap starting.
I20240503 08:12:31.232728 4006 tablet_service.cc:1467] Processing CreateTablet for tablet f727ac7c101f4375bf0a447c842a4ce2 (DEFAULT_TABLE table=test-workload [id=d4e73938d0bd4894a1b19c09e62f6a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.233055 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f727ac7c101f4375bf0a447c842a4ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.233227 4136 tablet_bootstrap.cc:654] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.234814 4136 tablet_bootstrap.cc:492] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146: No bootstrap required, opened a new log
I20240503 08:12:31.234973 4136 ts_tablet_manager.cc:1367] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:31.235653 4136 raft_consensus.cc:348] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd35f7bbd58d4202b068e45a23696146" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 35145 } } peers { permanent_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44451 } } peers { permanent_uuid: "982215fe60de4d1496b1919b1084c4c8" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 37571 } }
I20240503 08:12:31.235937 4136 raft_consensus.cc:374] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.236007 4136 raft_consensus.cc:729] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd35f7bbd58d4202b068e45a23696146, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.236218 4136 consensus_queue.cc:260] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd35f7bbd58d4202b068e45a23696146" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 35145 } } peers { permanent_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44451 } } peers { permanent_uuid: "982215fe60de4d1496b1919b1084c4c8" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 37571 } }
I20240503 08:12:31.236896 4136 ts_tablet_manager.cc:1398] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:31.242707 4137 tablet_bootstrap.cc:492] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30: Bootstrap starting.
I20240503 08:12:31.243146 4140 tablet_bootstrap.cc:492] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8: Bootstrap starting.
I20240503 08:12:31.244004 4137 tablet_bootstrap.cc:654] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.244417 4140 tablet_bootstrap.cc:654] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.245574 4137 tablet_bootstrap.cc:492] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30: No bootstrap required, opened a new log
I20240503 08:12:31.245640 4140 tablet_bootstrap.cc:492] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8: No bootstrap required, opened a new log
I20240503 08:12:31.245697 4137 ts_tablet_manager.cc:1367] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:31.245738 4140 ts_tablet_manager.cc:1367] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:31.246343 4140 raft_consensus.cc:348] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd35f7bbd58d4202b068e45a23696146" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 35145 } } peers { permanent_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44451 } } peers { permanent_uuid: "982215fe60de4d1496b1919b1084c4c8" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 37571 } }
I20240503 08:12:31.246343 4137 raft_consensus.cc:348] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd35f7bbd58d4202b068e45a23696146" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 35145 } } peers { permanent_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44451 } } peers { permanent_uuid: "982215fe60de4d1496b1919b1084c4c8" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 37571 } }
I20240503 08:12:31.246555 4140 raft_consensus.cc:374] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.246564 4137 raft_consensus.cc:374] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.246619 4140 raft_consensus.cc:729] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982215fe60de4d1496b1919b1084c4c8, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.246625 4137 raft_consensus.cc:729] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb00f793a4d40b4bacff7f5457d1c30, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.246840 4137 consensus_queue.cc:260] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd35f7bbd58d4202b068e45a23696146" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 35145 } } peers { permanent_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44451 } } peers { permanent_uuid: "982215fe60de4d1496b1919b1084c4c8" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 37571 } }
I20240503 08:12:31.246840 4140 consensus_queue.cc:260] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd35f7bbd58d4202b068e45a23696146" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 35145 } } peers { permanent_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44451 } } peers { permanent_uuid: "982215fe60de4d1496b1919b1084c4c8" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 37571 } }
I20240503 08:12:31.247582 4137 ts_tablet_manager.cc:1398] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:31.247699 4140 ts_tablet_manager.cc:1398] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:31.602980 4144 raft_consensus.cc:482] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:31.603154 4144 raft_consensus.cc:504] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd35f7bbd58d4202b068e45a23696146" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 35145 } } peers { permanent_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44451 } } peers { permanent_uuid: "982215fe60de4d1496b1919b1084c4c8" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 37571 } }
I20240503 08:12:31.603972 4144 leader_election.cc:288] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd35f7bbd58d4202b068e45a23696146 (127.0.230.1:35145), 982215fe60de4d1496b1919b1084c4c8 (127.0.230.2:37571)
I20240503 08:12:31.609037 3941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f727ac7c101f4375bf0a447c842a4ce2" candidate_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd35f7bbd58d4202b068e45a23696146" is_pre_election: true
I20240503 08:12:31.609105 4016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f727ac7c101f4375bf0a447c842a4ce2" candidate_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982215fe60de4d1496b1919b1084c4c8" is_pre_election: true
I20240503 08:12:31.609275 4016 raft_consensus.cc:2454] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffb00f793a4d40b4bacff7f5457d1c30 in term 0.
I20240503 08:12:31.609275 3941 raft_consensus.cc:2454] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffb00f793a4d40b4bacff7f5457d1c30 in term 0.
I20240503 08:12:31.609707 4053 leader_election.cc:302] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 982215fe60de4d1496b1919b1084c4c8, ffb00f793a4d40b4bacff7f5457d1c30; no voters:
I20240503 08:12:31.609951 4144 raft_consensus.cc:2781] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:31.610033 4144 raft_consensus.cc:482] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:31.610097 4144 raft_consensus.cc:3037] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.611837 4144 raft_consensus.cc:504] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd35f7bbd58d4202b068e45a23696146" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 35145 } } peers { permanent_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44451 } } peers { permanent_uuid: "982215fe60de4d1496b1919b1084c4c8" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 37571 } }
I20240503 08:12:31.612293 4144 leader_election.cc:288] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [CANDIDATE]: Term 1 election: Requested vote from peers cd35f7bbd58d4202b068e45a23696146 (127.0.230.1:35145), 982215fe60de4d1496b1919b1084c4c8 (127.0.230.2:37571)
I20240503 08:12:31.612645 3941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f727ac7c101f4375bf0a447c842a4ce2" candidate_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd35f7bbd58d4202b068e45a23696146"
I20240503 08:12:31.612690 4016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f727ac7c101f4375bf0a447c842a4ce2" candidate_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982215fe60de4d1496b1919b1084c4c8"
I20240503 08:12:31.612799 3941 raft_consensus.cc:3037] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.612838 4016 raft_consensus.cc:3037] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.614557 3941 raft_consensus.cc:2454] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffb00f793a4d40b4bacff7f5457d1c30 in term 1.
I20240503 08:12:31.614742 4016 raft_consensus.cc:2454] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffb00f793a4d40b4bacff7f5457d1c30 in term 1.
I20240503 08:12:31.614866 4056 leader_election.cc:302] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd35f7bbd58d4202b068e45a23696146, ffb00f793a4d40b4bacff7f5457d1c30; no voters:
I20240503 08:12:31.615082 4144 raft_consensus.cc:2781] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:31.615448 4144 raft_consensus.cc:686] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 1 LEADER]: Becoming Leader. State: Replica: ffb00f793a4d40b4bacff7f5457d1c30, State: Running, Role: LEADER
I20240503 08:12:31.615667 4144 consensus_queue.cc:237] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd35f7bbd58d4202b068e45a23696146" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 35145 } } peers { permanent_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44451 } } peers { permanent_uuid: "982215fe60de4d1496b1919b1084c4c8" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 37571 } }
I20240503 08:12:31.617740 3838 catalog_manager.cc:5484] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffb00f793a4d40b4bacff7f5457d1c30 (127.0.230.3). New cstate: current_term: 1 leader_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd35f7bbd58d4202b068e45a23696146" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 35145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffb00f793a4d40b4bacff7f5457d1c30" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 44451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "982215fe60de4d1496b1919b1084c4c8" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 37571 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:31.947105 3887 placement_policy_util.cc:407] tablet f727ac7c101f4375bf0a447c842a4ce2: detected majority of replicas (2 of 3) at location L0
I20240503 08:12:31.947574 3892 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:31.947949 3892 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:31.948041 3892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:32.140657 4149 consensus_queue.cc:1035] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd35f7bbd58d4202b068e45a23696146" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 35145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.142154 4149 consensus_queue.cc:1035] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982215fe60de4d1496b1919b1084c4c8" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 37571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.949797 3887 placement_policy_util.cc:407] tablet f727ac7c101f4375bf0a447c842a4ce2: detected majority of replicas (2 of 3) at location L0
I20240503 08:12:33.948386 3892 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:33.948837 3892 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:33.948937 3892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:33.952075 3887 placement_policy_util.cc:407] tablet f727ac7c101f4375bf0a447c842a4ce2: detected majority of replicas (2 of 3) at location L0
I20240503 08:12:34.954398 3887 placement_policy_util.cc:407] tablet f727ac7c101f4375bf0a447c842a4ce2: detected majority of replicas (2 of 3) at location L0
I20240503 08:12:35.949330 3892 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:35.949790 3892 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:35.949888 3892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:35.956763 3887 placement_policy_util.cc:407] tablet f727ac7c101f4375bf0a447c842a4ce2: detected majority of replicas (2 of 3) at location L0
I20240503 08:12:36.959120 3887 placement_policy_util.cc:407] tablet f727ac7c101f4375bf0a447c842a4ce2: detected majority of replicas (2 of 3) at location L0
I20240503 08:12:37.950246 3892 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:37.950706 3892 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:37.950815 3892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:37.961608 3887 placement_policy_util.cc:407] tablet f727ac7c101f4375bf0a447c842a4ce2: detected majority of replicas (2 of 3) at location L0
I20240503 08:12:38.963925 3887 placement_policy_util.cc:407] tablet f727ac7c101f4375bf0a447c842a4ce2: detected majority of replicas (2 of 3) at location L0
I20240503 08:12:39.951186 3892 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:39.951663 3892 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:39.951881 3892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:39.966291 3887 placement_policy_util.cc:407] tablet f727ac7c101f4375bf0a447c842a4ce2: detected majority of replicas (2 of 3) at location L0
I20240503 08:12:40.968735 3887 placement_policy_util.cc:407] tablet f727ac7c101f4375bf0a447c842a4ce2: detected majority of replicas (2 of 3) at location L0
I20240503 08:12:41.952292 3892 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:41.952769 3892 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:41.952872 3892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:41.971200 3887 placement_policy_util.cc:407] tablet f727ac7c101f4375bf0a447c842a4ce2: detected majority of replicas (2 of 3) at location L0
I20240503 08:12:42.739207 920 tablet_server.cc:178] TabletServer@127.0.230.1:0 shutting down...
I20240503 08:12:42.752375 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:42.752648 920 tablet_replica.cc:330] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146: stopping tablet replica
I20240503 08:12:42.752871 920 raft_consensus.cc:2229] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.753065 920 raft_consensus.cc:2258] T f727ac7c101f4375bf0a447c842a4ce2 P cd35f7bbd58d4202b068e45a23696146 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.766816 920 tablet_server.cc:194] TabletServer@127.0.230.1:0 shutdown complete.
I20240503 08:12:42.769654 920 tablet_server.cc:178] TabletServer@127.0.230.2:0 shutting down...
I20240503 08:12:42.777896 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:42.778151 920 tablet_replica.cc:330] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8: stopping tablet replica
I20240503 08:12:42.778333 920 raft_consensus.cc:2229] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.778527 920 raft_consensus.cc:2258] T f727ac7c101f4375bf0a447c842a4ce2 P 982215fe60de4d1496b1919b1084c4c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.791793 920 tablet_server.cc:194] TabletServer@127.0.230.2:0 shutdown complete.
I20240503 08:12:42.794579 920 tablet_server.cc:178] TabletServer@127.0.230.3:0 shutting down...
I20240503 08:12:42.803103 920 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:42.803336 920 tablet_replica.cc:330] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30: stopping tablet replica
I20240503 08:12:42.803534 920 raft_consensus.cc:2229] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:42.803828 920 raft_consensus.cc:2258] T f727ac7c101f4375bf0a447c842a4ce2 P ffb00f793a4d40b4bacff7f5457d1c30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.818606 920 tablet_server.cc:194] TabletServer@127.0.230.3:0 shutdown complete.
I20240503 08:12:42.821298 920 master.cc:537] Master@127.0.230.62:40169 shutting down...
I20240503 08:12:42.828104 920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:42.828315 920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.828404 920 tablet_replica.cc:330] T 00000000000000000000000000000000 P ee05e48b71004d8bbb955f26ac0b7b56: stopping tablet replica
I20240503 08:12:42.848106 920 master.cc:552] Master@127.0.230.62:40169 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12968 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240503 08:12:42.858063 920 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.230.62:44605
I20240503 08:12:42.858521 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:42.861065 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:42.861641 4161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:42.862087 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:42.862746 920 server_base.cc:1034] running on GCE node
I20240503 08:12:42.862928 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:42.862973 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:42.862999 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723962862991 us; error 0 us; skew 500 ppm
I20240503 08:12:42.863126 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:42.869098 920 webserver.cc:460] Webserver started at http://127.0.230.62:40485/ using document root <none> and password file <none>
I20240503 08:12:42.869303 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:42.869359 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:42.869473 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:42.870074 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/master-0-root/instance:
uuid: "e6454db76dd0484a98f46cc73e746430"
format_stamp: "Formatted at 2024-05-03 08:12:42 on dist-test-slave-2jf9"
I20240503 08:12:42.872397 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:42.873782 4168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:42.874109 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:42.874192 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/master-0-root
uuid: "e6454db76dd0484a98f46cc73e746430"
format_stamp: "Formatted at 2024-05-03 08:12:42 on dist-test-slave-2jf9"
I20240503 08:12:42.874275 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:42.885207 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:42.885588 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:42.894634 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.62:44605
I20240503 08:12:42.894650 4219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.62:44605 every 8 connection(s)
I20240503 08:12:42.895857 4220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.899720 4220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430: Bootstrap starting.
I20240503 08:12:42.900749 4220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.902027 4220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430: No bootstrap required, opened a new log
I20240503 08:12:42.902534 4220 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6454db76dd0484a98f46cc73e746430" member_type: VOTER }
I20240503 08:12:42.902655 4220 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.902710 4220 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6454db76dd0484a98f46cc73e746430, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.902885 4220 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6454db76dd0484a98f46cc73e746430" member_type: VOTER }
I20240503 08:12:42.902988 4220 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:12:42.903038 4220 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:12:42.903100 4220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.904747 4220 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6454db76dd0484a98f46cc73e746430" member_type: VOTER }
I20240503 08:12:42.904922 4220 leader_election.cc:302] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6454db76dd0484a98f46cc73e746430; no voters:
I20240503 08:12:42.905241 4220 leader_election.cc:288] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:12:42.905324 4223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:42.905740 4223 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [term 1 LEADER]: Becoming Leader. State: Replica: e6454db76dd0484a98f46cc73e746430, State: Running, Role: LEADER
I20240503 08:12:42.905928 4223 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6454db76dd0484a98f46cc73e746430" member_type: VOTER }
I20240503 08:12:42.906006 4220 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:12:42.906528 4224 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6454db76dd0484a98f46cc73e746430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6454db76dd0484a98f46cc73e746430" member_type: VOTER } }
I20240503 08:12:42.906572 4225 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6454db76dd0484a98f46cc73e746430. Latest consensus state: current_term: 1 leader_uuid: "e6454db76dd0484a98f46cc73e746430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6454db76dd0484a98f46cc73e746430" member_type: VOTER } }
I20240503 08:12:42.906702 4224 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:42.906787 4225 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:42.907651 4228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:12:42.909322 4228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:12:42.910250 920 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:12:42.912050 4228 catalog_manager.cc:1347] Generated new cluster ID: e85961a9d6a54bf4adbd47e529113b6a
I20240503 08:12:42.912123 4228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:12:42.923892 4228 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:12:42.924600 4228 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:12:42.929986 4228 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e6454db76dd0484a98f46cc73e746430: Generated new TSK 0
I20240503 08:12:42.930181 4228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:12:42.942896 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:42.945916 4242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:42.946115 4241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:42.946476 4244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:42.947126 920 server_base.cc:1034] running on GCE node
I20240503 08:12:42.947342 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:42.947419 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:42.947459 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723962947456 us; error 0 us; skew 500 ppm
I20240503 08:12:42.947583 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:42.948661 920 webserver.cc:460] Webserver started at http://127.0.230.1:37963/ using document root <none> and password file <none>
I20240503 08:12:42.948840 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:42.948886 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:42.948985 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:42.949568 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-0-root/instance:
uuid: "8df53dcdf65b407480e6d32372b48a35"
format_stamp: "Formatted at 2024-05-03 08:12:42 on dist-test-slave-2jf9"
I20240503 08:12:42.951838 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 08:12:42.953253 4249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:42.953573 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:42.953662 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-0-root
uuid: "8df53dcdf65b407480e6d32372b48a35"
format_stamp: "Formatted at 2024-05-03 08:12:42 on dist-test-slave-2jf9"
I20240503 08:12:42.953765 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:42.969542 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:42.969995 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:42.970453 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:42.971244 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:42.971294 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:42.971355 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:42.971421 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:42.981390 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.1:32947
I20240503 08:12:42.981402 4311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.1:32947 every 8 connection(s)
I20240503 08:12:42.982884 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:42.985916 4316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:42.986295 4317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:42.987118 4319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:42.987149 920 server_base.cc:1034] running on GCE node
I20240503 08:12:42.987500 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:42.987556 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:42.987586 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723962987582 us; error 0 us; skew 500 ppm
I20240503 08:12:42.987720 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:42.988899 4312 heartbeater.cc:347] Connected to a master server at 127.0.230.62:44605
I20240503 08:12:42.988991 920 webserver.cc:460] Webserver started at http://127.0.230.2:34547/ using document root <none> and password file <none>
I20240503 08:12:42.989015 4312 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:42.989210 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:42.989269 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:42.989358 4312 heartbeater.cc:511] Master 127.0.230.62:44605 requested a full tablet report, sending...
I20240503 08:12:42.989373 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:42.989984 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-1-root/instance:
uuid: "3482c85dd194402bbc0038963ed16d35"
format_stamp: "Formatted at 2024-05-03 08:12:42 on dist-test-slave-2jf9"
I20240503 08:12:42.990145 4185 ts_manager.cc:194] Registered new tserver with Master: 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947)
I20240503 08:12:42.991106 4185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56828
I20240503 08:12:42.992317 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:42.993726 4324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:42.994050 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:42.994129 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-1-root
uuid: "3482c85dd194402bbc0038963ed16d35"
format_stamp: "Formatted at 2024-05-03 08:12:42 on dist-test-slave-2jf9"
I20240503 08:12:42.994208 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:43.015657 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:43.016112 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:43.016559 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:43.017295 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:43.017346 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:43.017406 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:43.017444 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:43.027145 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.2:44611
I20240503 08:12:43.027165 4386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.2:44611 every 8 connection(s)
I20240503 08:12:43.028493 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:43.031664 4391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:43.031864 4390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:43.032217 4393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:43.032737 920 server_base.cc:1034] running on GCE node
I20240503 08:12:43.033025 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:43.033121 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:43.033164 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723963033160 us; error 0 us; skew 500 ppm
I20240503 08:12:43.033299 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:43.034530 4387 heartbeater.cc:347] Connected to a master server at 127.0.230.62:44605
I20240503 08:12:43.034557 920 webserver.cc:460] Webserver started at http://127.0.230.3:45377/ using document root <none> and password file <none>
I20240503 08:12:43.034646 4387 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:43.034768 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:43.034829 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:43.034932 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:43.034962 4387 heartbeater.cc:511] Master 127.0.230.62:44605 requested a full tablet report, sending...
I20240503 08:12:43.035595 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-2-root/instance:
uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
format_stamp: "Formatted at 2024-05-03 08:12:43 on dist-test-slave-2jf9"
I20240503 08:12:43.035760 4185 ts_manager.cc:194] Registered new tserver with Master: 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:43.036427 4185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56832
I20240503 08:12:43.037925 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:43.039271 4398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:43.039559 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 08:12:43.039645 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-2-root
uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
format_stamp: "Formatted at 2024-05-03 08:12:43 on dist-test-slave-2jf9"
I20240503 08:12:43.039734 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:43.052970 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:43.053428 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:43.053896 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:43.054636 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:43.054687 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:43.054747 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:43.054780 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:43.064473 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.3:38567
I20240503 08:12:43.064517 4460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.3:38567 every 8 connection(s)
I20240503 08:12:43.070317 4461 heartbeater.cc:347] Connected to a master server at 127.0.230.62:44605
I20240503 08:12:43.070432 4461 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:43.070719 4461 heartbeater.cc:511] Master 127.0.230.62:44605 requested a full tablet report, sending...
I20240503 08:12:43.071489 4185 ts_manager.cc:194] Registered new tserver with Master: 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:43.071816 920 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006609856s
I20240503 08:12:43.072206 4185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56836
I20240503 08:12:43.992273 4312 heartbeater.cc:503] Master 127.0.230.62:44605 was elected leader, sending a full tablet report...
I20240503 08:12:44.037541 4387 heartbeater.cc:503] Master 127.0.230.62:44605 was elected leader, sending a full tablet report...
I20240503 08:12:44.073457 4461 heartbeater.cc:503] Master 127.0.230.62:44605 was elected leader, sending a full tablet report...
I20240503 08:12:44.098984 920 test_util.cc:268] Using random seed: 1220843938
I20240503 08:12:44.110316 4185 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56840:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:12:44.111061 4185 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:12:44.136720 4277 tablet_service.cc:1467] Processing CreateTablet for tablet 1440f42da8514e618147c69fa2795415 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 08:12:44.136852 4276 tablet_service.cc:1467] Processing CreateTablet for tablet b82e1e5d27e441ccbd62d1296c06a2f4 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 08:12:44.137239 4277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1440f42da8514e618147c69fa2795415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.137285 4271 tablet_service.cc:1467] Processing CreateTablet for tablet 374e45fb23af484aa3d9c6da1f1863d3 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 08:12:44.137421 4276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82e1e5d27e441ccbd62d1296c06a2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.137671 4271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374e45fb23af484aa3d9c6da1f1863d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.138379 4425 tablet_service.cc:1467] Processing CreateTablet for tablet b82e1e5d27e441ccbd62d1296c06a2f4 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 08:12:44.138664 4274 tablet_service.cc:1467] Processing CreateTablet for tablet 2d0dc42483a64c90a1492a7353e2ba4f (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 08:12:44.138773 4425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82e1e5d27e441ccbd62d1296c06a2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.139006 4274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d0dc42483a64c90a1492a7353e2ba4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.143618 4270 tablet_service.cc:1467] Processing CreateTablet for tablet f8ed061642594372a70df1d766d7b646 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 08:12:44.143743 4273 tablet_service.cc:1467] Processing CreateTablet for tablet b2b69fcfaa7742919bc05385b49623dd (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 08:12:44.144029 4270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ed061642594372a70df1d766d7b646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.144364 4272 tablet_service.cc:1467] Processing CreateTablet for tablet d7037feb51534b57b0ac70344cdc1e18 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 08:12:44.144765 4272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7037feb51534b57b0ac70344cdc1e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.148452 4426 tablet_service.cc:1467] Processing CreateTablet for tablet 1440f42da8514e618147c69fa2795415 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 08:12:44.148892 4426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1440f42da8514e618147c69fa2795415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.151190 4271 tablet_service.cc:1467] Processing CreateTablet for tablet c94415b8c11d4f9790344c93eff4c331 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 08:12:44.151543 4271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94415b8c11d4f9790344c93eff4c331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.138379 4275 tablet_service.cc:1467] Processing CreateTablet for tablet 03e50d2db8824a11af7a5e4f9f3ed240 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 08:12:44.152360 4275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e50d2db8824a11af7a5e4f9f3ed240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.154213 4273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b69fcfaa7742919bc05385b49623dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.151850 4270 tablet_service.cc:1467] Processing CreateTablet for tablet bd99f1526db74ef5ace0a312d7afe495 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 08:12:44.159044 4270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd99f1526db74ef5ace0a312d7afe495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.163609 4424 tablet_service.cc:1467] Processing CreateTablet for tablet 03e50d2db8824a11af7a5e4f9f3ed240 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 08:12:44.164049 4424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e50d2db8824a11af7a5e4f9f3ed240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.167838 4269 tablet_service.cc:1467] Processing CreateTablet for tablet ae715a227487424dbe08825a7f7d9819 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 08:12:44.168102 4423 tablet_service.cc:1467] Processing CreateTablet for tablet 2d0dc42483a64c90a1492a7353e2ba4f (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 08:12:44.168275 4269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae715a227487424dbe08825a7f7d9819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.168530 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d0dc42483a64c90a1492a7353e2ba4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.172029 4422 tablet_service.cc:1467] Processing CreateTablet for tablet b2b69fcfaa7742919bc05385b49623dd (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 08:12:44.172503 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b69fcfaa7742919bc05385b49623dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.175495 4269 tablet_service.cc:1467] Processing CreateTablet for tablet 59aedfe83feb48d799de8eda2afb1860 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 08:12:44.175947 4269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59aedfe83feb48d799de8eda2afb1860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.176012 4421 tablet_service.cc:1467] Processing CreateTablet for tablet d7037feb51534b57b0ac70344cdc1e18 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 08:12:44.176419 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7037feb51534b57b0ac70344cdc1e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.179764 4420 tablet_service.cc:1467] Processing CreateTablet for tablet 374e45fb23af484aa3d9c6da1f1863d3 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 08:12:44.180132 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374e45fb23af484aa3d9c6da1f1863d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.183768 4419 tablet_service.cc:1467] Processing CreateTablet for tablet f8ed061642594372a70df1d766d7b646 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 08:12:44.184175 4419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ed061642594372a70df1d766d7b646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.187125 4422 tablet_service.cc:1467] Processing CreateTablet for tablet ae715a227487424dbe08825a7f7d9819 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 08:12:44.187805 4418 tablet_service.cc:1467] Processing CreateTablet for tablet bd99f1526db74ef5ace0a312d7afe495 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 08:12:44.188396 4482 tablet_bootstrap.cc:492] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed: Bootstrap starting.
I20240503 08:12:44.188664 4481 tablet_bootstrap.cc:492] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35: Bootstrap starting.
I20240503 08:12:44.188597 4417 tablet_service.cc:1467] Processing CreateTablet for tablet c94415b8c11d4f9790344c93eff4c331 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 08:12:44.190418 4481 tablet_bootstrap.cc:654] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.190665 4423 tablet_service.cc:1467] Processing CreateTablet for tablet 59aedfe83feb48d799de8eda2afb1860 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 08:12:44.193508 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae715a227487424dbe08825a7f7d9819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.193930 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59aedfe83feb48d799de8eda2afb1860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.195752 4417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94415b8c11d4f9790344c93eff4c331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.196314 4418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd99f1526db74ef5ace0a312d7afe495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.197376 4481 tablet_bootstrap.cc:492] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35: No bootstrap required, opened a new log
I20240503 08:12:44.201387 4482 tablet_bootstrap.cc:654] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.200901 4481 ts_tablet_manager.cc:1367] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240503 08:12:44.207603 4481 raft_consensus.cc:348] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.207826 4481 raft_consensus.cc:374] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.207931 4481 raft_consensus.cc:729] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.208184 4481 consensus_queue.cc:260] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.209416 4481 ts_tablet_manager.cc:1398] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:44.209750 4481 tablet_bootstrap.cc:492] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35: Bootstrap starting.
I20240503 08:12:44.210179 4482 tablet_bootstrap.cc:492] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed: No bootstrap required, opened a new log
I20240503 08:12:44.210079 4350 tablet_service.cc:1467] Processing CreateTablet for tablet 03e50d2db8824a11af7a5e4f9f3ed240 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 08:12:44.210364 4482 ts_tablet_manager.cc:1367] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.005s
I20240503 08:12:44.210696 4350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e50d2db8824a11af7a5e4f9f3ed240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.211036 4352 tablet_service.cc:1467] Processing CreateTablet for tablet 1440f42da8514e618147c69fa2795415 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 08:12:44.211118 4482 raft_consensus.cc:348] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.211319 4482 raft_consensus.cc:374] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.211418 4482 raft_consensus.cc:729] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.211501 4481 tablet_bootstrap.cc:654] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.211686 4482 consensus_queue.cc:260] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.212051 4351 tablet_service.cc:1467] Processing CreateTablet for tablet b82e1e5d27e441ccbd62d1296c06a2f4 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 08:12:44.212877 4349 tablet_service.cc:1467] Processing CreateTablet for tablet 2d0dc42483a64c90a1492a7353e2ba4f (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 08:12:44.213665 4348 tablet_service.cc:1467] Processing CreateTablet for tablet b2b69fcfaa7742919bc05385b49623dd (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 08:12:44.214412 4347 tablet_service.cc:1467] Processing CreateTablet for tablet d7037feb51534b57b0ac70344cdc1e18 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 08:12:44.215242 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 374e45fb23af484aa3d9c6da1f1863d3 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 08:12:44.216022 4345 tablet_service.cc:1467] Processing CreateTablet for tablet f8ed061642594372a70df1d766d7b646 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 08:12:44.216336 4481 tablet_bootstrap.cc:492] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35: No bootstrap required, opened a new log
I20240503 08:12:44.216481 4481 ts_tablet_manager.cc:1367] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240503 08:12:44.216825 4344 tablet_service.cc:1467] Processing CreateTablet for tablet bd99f1526db74ef5ace0a312d7afe495 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 08:12:44.217459 4481 raft_consensus.cc:348] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.217519 4343 tablet_service.cc:1467] Processing CreateTablet for tablet c94415b8c11d4f9790344c93eff4c331 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 08:12:44.218027 4481 raft_consensus.cc:374] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.218233 4481 raft_consensus.cc:729] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.219142 4352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1440f42da8514e618147c69fa2795415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.218686 4481 consensus_queue.cc:260] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.221665 4481 ts_tablet_manager.cc:1398] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 08:12:44.223951 4344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd99f1526db74ef5ace0a312d7afe495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.223958 4481 tablet_bootstrap.cc:492] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35: Bootstrap starting.
I20240503 08:12:44.224182 4343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94415b8c11d4f9790344c93eff4c331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.224380 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b69fcfaa7742919bc05385b49623dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.227313 4345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ed061642594372a70df1d766d7b646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.230319 4343 tablet_service.cc:1467] Processing CreateTablet for tablet ae715a227487424dbe08825a7f7d9819 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 08:12:44.230645 4343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae715a227487424dbe08825a7f7d9819. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.231587 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374e45fb23af484aa3d9c6da1f1863d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.235558 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7037feb51534b57b0ac70344cdc1e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.236953 4481 tablet_bootstrap.cc:654] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.238147 4482 ts_tablet_manager.cc:1398] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent starting tablet: real 0.028s user 0.016s sys 0.012s
I20240503 08:12:44.238420 4482 tablet_bootstrap.cc:492] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed: Bootstrap starting.
I20240503 08:12:44.239993 4482 tablet_bootstrap.cc:654] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.240219 4343 tablet_service.cc:1467] Processing CreateTablet for tablet 59aedfe83feb48d799de8eda2afb1860 (DEFAULT_TABLE table=test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 08:12:44.240581 4343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59aedfe83feb48d799de8eda2afb1860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.240880 4349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d0dc42483a64c90a1492a7353e2ba4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.243595 4351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82e1e5d27e441ccbd62d1296c06a2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.245380 4481 tablet_bootstrap.cc:492] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35: No bootstrap required, opened a new log
I20240503 08:12:44.245522 4481 ts_tablet_manager.cc:1367] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20240503 08:12:44.246177 4481 raft_consensus.cc:348] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.246361 4481 raft_consensus.cc:374] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.246438 4481 raft_consensus.cc:729] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.246688 4481 consensus_queue.cc:260] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.247154 4488 tablet_bootstrap.cc:492] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35: Bootstrap starting.
I20240503 08:12:44.249047 4488 tablet_bootstrap.cc:654] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.250346 4481 ts_tablet_manager.cc:1398] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 08:12:44.251912 4481 tablet_bootstrap.cc:492] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35: Bootstrap starting.
I20240503 08:12:44.253734 4481 tablet_bootstrap.cc:654] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.258520 4488 tablet_bootstrap.cc:492] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35: No bootstrap required, opened a new log
I20240503 08:12:44.258718 4488 ts_tablet_manager.cc:1367] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 08:12:44.259482 4488 raft_consensus.cc:348] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.259698 4488 raft_consensus.cc:374] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.259794 4488 raft_consensus.cc:729] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.260068 4488 consensus_queue.cc:260] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.260457 4482 tablet_bootstrap.cc:492] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed: No bootstrap required, opened a new log
I20240503 08:12:44.260658 4482 ts_tablet_manager.cc:1367] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.011s
I20240503 08:12:44.261451 4481 tablet_bootstrap.cc:492] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35: No bootstrap required, opened a new log
I20240503 08:12:44.261413 4482 raft_consensus.cc:348] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.261595 4481 ts_tablet_manager.cc:1367] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240503 08:12:44.261713 4482 raft_consensus.cc:374] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.261868 4482 raft_consensus.cc:729] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.262149 4482 consensus_queue.cc:260] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.262300 4481 raft_consensus.cc:348] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.262490 4481 raft_consensus.cc:374] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.262599 4481 raft_consensus.cc:729] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.262910 4482 ts_tablet_manager.cc:1398] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:44.262877 4481 consensus_queue.cc:260] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.263224 4482 tablet_bootstrap.cc:492] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed: Bootstrap starting.
I20240503 08:12:44.263957 4481 ts_tablet_manager.cc:1398] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:44.264279 4481 tablet_bootstrap.cc:492] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35: Bootstrap starting.
I20240503 08:12:44.264653 4482 tablet_bootstrap.cc:654] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.265779 4481 tablet_bootstrap.cc:654] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.269410 4481 tablet_bootstrap.cc:492] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35: No bootstrap required, opened a new log
I20240503 08:12:44.269620 4481 ts_tablet_manager.cc:1367] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:12:44.270406 4481 raft_consensus.cc:348] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.270701 4481 raft_consensus.cc:374] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.270857 4481 raft_consensus.cc:729] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.271142 4481 consensus_queue.cc:260] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.271829 4481 ts_tablet_manager.cc:1398] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:44.272163 4481 tablet_bootstrap.cc:492] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35: Bootstrap starting.
I20240503 08:12:44.273845 4481 tablet_bootstrap.cc:654] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.274400 4488 ts_tablet_manager.cc:1398] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20240503 08:12:44.275481 4488 tablet_bootstrap.cc:492] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35: Bootstrap starting.
I20240503 08:12:44.276866 4488 tablet_bootstrap.cc:654] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.278389 4488 tablet_bootstrap.cc:492] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35: No bootstrap required, opened a new log
I20240503 08:12:44.278524 4488 ts_tablet_manager.cc:1367] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:44.278602 4482 tablet_bootstrap.cc:492] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed: No bootstrap required, opened a new log
I20240503 08:12:44.278769 4482 ts_tablet_manager.cc:1367] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240503 08:12:44.279134 4488 raft_consensus.cc:348] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.279318 4488 raft_consensus.cc:374] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.279412 4488 raft_consensus.cc:729] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.279506 4482 raft_consensus.cc:348] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.279708 4482 raft_consensus.cc:374] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.279626 4488 consensus_queue.cc:260] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.279798 4482 raft_consensus.cc:729] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.280118 4488 ts_tablet_manager.cc:1398] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:44.280033 4482 consensus_queue.cc:260] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.280377 4488 tablet_bootstrap.cc:492] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35: Bootstrap starting.
I20240503 08:12:44.280645 4482 ts_tablet_manager.cc:1398] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:44.280916 4482 tablet_bootstrap.cc:492] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed: Bootstrap starting.
I20240503 08:12:44.281841 4488 tablet_bootstrap.cc:654] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.282538 4482 tablet_bootstrap.cc:654] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.283630 4481 tablet_bootstrap.cc:492] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35: No bootstrap required, opened a new log
I20240503 08:12:44.283763 4481 ts_tablet_manager.cc:1367] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20240503 08:12:44.284391 4481 raft_consensus.cc:348] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.284605 4481 raft_consensus.cc:374] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.284701 4481 raft_consensus.cc:729] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.284910 4481 consensus_queue.cc:260] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.285288 4482 tablet_bootstrap.cc:492] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed: No bootstrap required, opened a new log
I20240503 08:12:44.285370 4481 ts_tablet_manager.cc:1398] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:44.285418 4482 ts_tablet_manager.cc:1367] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240503 08:12:44.285954 4481 tablet_bootstrap.cc:492] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35: Bootstrap starting.
I20240503 08:12:44.286046 4482 raft_consensus.cc:348] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.286211 4482 raft_consensus.cc:374] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.286283 4482 raft_consensus.cc:729] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.286506 4482 consensus_queue.cc:260] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.286988 4482 ts_tablet_manager.cc:1398] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:44.287231 4482 tablet_bootstrap.cc:492] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed: Bootstrap starting.
I20240503 08:12:44.287499 4481 tablet_bootstrap.cc:654] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.288656 4482 tablet_bootstrap.cc:654] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.291476 4488 tablet_bootstrap.cc:492] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35: No bootstrap required, opened a new log
I20240503 08:12:44.291611 4488 ts_tablet_manager.cc:1367] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240503 08:12:44.292286 4488 raft_consensus.cc:348] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.292475 4488 raft_consensus.cc:374] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.292554 4488 raft_consensus.cc:729] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.292822 4488 consensus_queue.cc:260] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.293296 4488 ts_tablet_manager.cc:1398] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:44.293562 4488 tablet_bootstrap.cc:492] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35: Bootstrap starting.
I20240503 08:12:44.294952 4488 tablet_bootstrap.cc:654] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.297820 4482 tablet_bootstrap.cc:492] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed: No bootstrap required, opened a new log
I20240503 08:12:44.297909 4481 tablet_bootstrap.cc:492] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35: No bootstrap required, opened a new log
I20240503 08:12:44.297967 4482 ts_tablet_manager.cc:1367] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240503 08:12:44.298028 4481 ts_tablet_manager.cc:1367] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240503 08:12:44.298318 4488 tablet_bootstrap.cc:492] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35: No bootstrap required, opened a new log
I20240503 08:12:44.298444 4488 ts_tablet_manager.cc:1367] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:12:44.298632 4482 raft_consensus.cc:348] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.298651 4481 raft_consensus.cc:348] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.298849 4482 raft_consensus.cc:374] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.298852 4481 raft_consensus.cc:374] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.298944 4482 raft_consensus.cc:729] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.298955 4481 raft_consensus.cc:729] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.299010 4488 raft_consensus.cc:348] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.299208 4488 raft_consensus.cc:374] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.299341 4488 raft_consensus.cc:729] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.299196 4482 consensus_queue.cc:260] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.299204 4481 consensus_queue.cc:260] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.299585 4488 consensus_queue.cc:260] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.299755 4482 ts_tablet_manager.cc:1398] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:44.300002 4482 tablet_bootstrap.cc:492] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed: Bootstrap starting.
I20240503 08:12:44.300029 4488 ts_tablet_manager.cc:1398] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:12:44.300277 4488 tablet_bootstrap.cc:492] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35: Bootstrap starting.
I20240503 08:12:44.301658 4482 tablet_bootstrap.cc:654] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.301841 4481 ts_tablet_manager.cc:1398] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:44.301812 4488 tablet_bootstrap.cc:654] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.304049 4481 tablet_bootstrap.cc:492] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35: Bootstrap starting.
I20240503 08:12:44.305578 4481 tablet_bootstrap.cc:654] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.306716 4488 tablet_bootstrap.cc:492] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35: No bootstrap required, opened a new log
I20240503 08:12:44.306874 4488 ts_tablet_manager.cc:1367] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240503 08:12:44.307252 4482 tablet_bootstrap.cc:492] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed: No bootstrap required, opened a new log
I20240503 08:12:44.307380 4481 tablet_bootstrap.cc:492] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35: No bootstrap required, opened a new log
I20240503 08:12:44.307417 4482 ts_tablet_manager.cc:1367] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 08:12:44.307564 4481 ts_tablet_manager.cc:1367] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:44.307657 4488 raft_consensus.cc:348] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.307857 4488 raft_consensus.cc:374] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.307941 4488 raft_consensus.cc:729] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.308197 4482 raft_consensus.cc:348] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.308226 4481 raft_consensus.cc:348] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.308193 4488 consensus_queue.cc:260] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.308451 4481 raft_consensus.cc:374] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.308429 4482 raft_consensus.cc:374] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.308526 4481 raft_consensus.cc:729] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.308533 4482 raft_consensus.cc:729] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.308743 4481 consensus_queue.cc:260] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.308796 4482 consensus_queue.cc:260] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.309345 4482 ts_tablet_manager.cc:1398] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:44.309597 4482 tablet_bootstrap.cc:492] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed: Bootstrap starting.
I20240503 08:12:44.310171 4481 ts_tablet_manager.cc:1398] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:44.310417 4481 tablet_bootstrap.cc:492] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35: Bootstrap starting.
I20240503 08:12:44.311157 4482 tablet_bootstrap.cc:654] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.311223 4488 ts_tablet_manager.cc:1398] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:44.311928 4481 tablet_bootstrap.cc:654] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.312285 4488 tablet_bootstrap.cc:492] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35: Bootstrap starting.
I20240503 08:12:44.313854 4488 tablet_bootstrap.cc:654] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.315819 4482 tablet_bootstrap.cc:492] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed: No bootstrap required, opened a new log
I20240503 08:12:44.315994 4482 ts_tablet_manager.cc:1367] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20240503 08:12:44.316490 4481 tablet_bootstrap.cc:492] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35: No bootstrap required, opened a new log
I20240503 08:12:44.316615 4481 ts_tablet_manager.cc:1367] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:44.316679 4482 raft_consensus.cc:348] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.316910 4482 raft_consensus.cc:374] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.317003 4482 raft_consensus.cc:729] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.317052 4488 tablet_bootstrap.cc:492] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35: No bootstrap required, opened a new log
I20240503 08:12:44.317168 4488 ts_tablet_manager.cc:1367] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 08:12:44.317206 4481 raft_consensus.cc:348] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.317229 4482 consensus_queue.cc:260] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.317396 4481 raft_consensus.cc:374] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.317787 4482 ts_tablet_manager.cc:1398] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:44.317821 4488 raft_consensus.cc:348] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.317546 4481 raft_consensus.cc:729] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.317991 4488 raft_consensus.cc:374] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.318069 4482 tablet_bootstrap.cc:492] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed: Bootstrap starting.
I20240503 08:12:44.318071 4488 raft_consensus.cc:729] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.318318 4488 consensus_queue.cc:260] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.318881 4488 ts_tablet_manager.cc:1398] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:44.319154 4488 tablet_bootstrap.cc:492] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35: Bootstrap starting.
I20240503 08:12:44.319593 4482 tablet_bootstrap.cc:654] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.319554 4481 consensus_queue.cc:260] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.320663 4488 tablet_bootstrap.cc:654] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.321031 4481 ts_tablet_manager.cc:1398] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:44.321362 4481 tablet_bootstrap.cc:492] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35: Bootstrap starting.
I20240503 08:12:44.321956 4486 raft_consensus.cc:482] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.322134 4486 raft_consensus.cc:504] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.323015 4481 tablet_bootstrap.cc:654] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.323242 4486 leader_election.cc:288] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:44.325525 4494 raft_consensus.cc:482] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.325673 4494 raft_consensus.cc:504] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.326622 4494 leader_election.cc:288] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.338387 4488 tablet_bootstrap.cc:492] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35: No bootstrap required, opened a new log
I20240503 08:12:44.338572 4488 ts_tablet_manager.cc:1367] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20240503 08:12:44.339373 4488 raft_consensus.cc:348] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.339617 4488 raft_consensus.cc:374] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.339701 4488 raft_consensus.cc:729] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.339920 4488 consensus_queue.cc:260] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.340528 4488 ts_tablet_manager.cc:1398] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:44.340823 4488 tablet_bootstrap.cc:492] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35: Bootstrap starting.
I20240503 08:12:44.342574 4488 tablet_bootstrap.cc:654] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.349640 4494 raft_consensus.cc:482] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.349785 4494 raft_consensus.cc:504] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.350386 4494 leader_election.cc:288] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.351104 4485 raft_consensus.cc:482] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.351243 4485 raft_consensus.cc:504] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.352224 4485 leader_election.cc:288] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.356057 4482 tablet_bootstrap.cc:492] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed: No bootstrap required, opened a new log
I20240503 08:12:44.356225 4482 ts_tablet_manager.cc:1367] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.005s
I20240503 08:12:44.356952 4482 raft_consensus.cc:348] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.357156 4482 raft_consensus.cc:374] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.357241 4482 raft_consensus.cc:729] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.357493 4482 consensus_queue.cc:260] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.358006 4482 ts_tablet_manager.cc:1398] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:44.358268 4482 tablet_bootstrap.cc:492] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed: Bootstrap starting.
I20240503 08:12:44.362112 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35" is_pre_election: true
I20240503 08:12:44.362433 4362 raft_consensus.cc:2454] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 0.
I20240503 08:12:44.368508 4481 tablet_bootstrap.cc:492] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35: No bootstrap required, opened a new log
I20240503 08:12:44.368708 4481 ts_tablet_manager.cc:1367] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35: Time spent bootstrapping tablet: real 0.047s user 0.035s sys 0.010s
I20240503 08:12:44.369100 4482 tablet_bootstrap.cc:654] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.369552 4481 raft_consensus.cc:348] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.369788 4481 raft_consensus.cc:374] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.369908 4481 raft_consensus.cc:729] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.370169 4481 consensus_queue.cc:260] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.370769 4481 ts_tablet_manager.cc:1398] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:44.371169 4481 tablet_bootstrap.cc:492] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35: Bootstrap starting.
I20240503 08:12:44.372851 4400 leader_election.cc:302] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 3523cd896bb842d68fbb6c15cdd1c3ed; no voters:
I20240503 08:12:44.373071 4481 tablet_bootstrap.cc:654] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.373157 4486 raft_consensus.cc:2781] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.373261 4486 raft_consensus.cc:482] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.373373 4486 raft_consensus.cc:3037] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.376597 4486 raft_consensus.cc:504] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.379971 4486 leader_election.cc:288] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:44.380406 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:44.380631 4362 raft_consensus.cc:3037] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.381525 4481 tablet_bootstrap.cc:492] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35: No bootstrap required, opened a new log
I20240503 08:12:44.381721 4481 ts_tablet_manager.cc:1367] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240503 08:12:44.382513 4481 raft_consensus.cc:348] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.382805 4481 raft_consensus.cc:374] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.382926 4481 raft_consensus.cc:729] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.383214 4481 consensus_queue.cc:260] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.383913 4362 raft_consensus.cc:2454] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 1.
I20240503 08:12:44.384289 4400 leader_election.cc:302] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 3523cd896bb842d68fbb6c15cdd1c3ed; no voters:
I20240503 08:12:44.384599 4481 ts_tablet_manager.cc:1398] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:44.384809 4486 raft_consensus.cc:2781] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.384963 4481 tablet_bootstrap.cc:492] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35: Bootstrap starting.
I20240503 08:12:44.386581 4481 tablet_bootstrap.cc:654] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.388685 4287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35" is_pre_election: true
I20240503 08:12:44.389025 4287 raft_consensus.cc:2454] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 0.
I20240503 08:12:44.389461 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:44.389725 4286 raft_consensus.cc:3037] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.393064 4286 raft_consensus.cc:2454] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 1.
I20240503 08:12:44.393955 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd99f1526db74ef5ace0a312d7afe495" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35" is_pre_election: true
I20240503 08:12:44.394184 4286 raft_consensus.cc:2454] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 0.
I20240503 08:12:44.394464 4287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7037feb51534b57b0ac70344cdc1e18" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35" is_pre_election: true
I20240503 08:12:44.394661 4287 raft_consensus.cc:2454] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 0.
I20240503 08:12:44.396168 4482 tablet_bootstrap.cc:492] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed: No bootstrap required, opened a new log
I20240503 08:12:44.396168 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd99f1526db74ef5ace0a312d7afe495" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" is_pre_election: true
I20240503 08:12:44.396139 4325 leader_election.cc:302] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.396626 4494 raft_consensus.cc:2781] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.396750 4494 raft_consensus.cc:482] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.396864 4494 raft_consensus.cc:3037] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.397271 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7037feb51534b57b0ac70344cdc1e18" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" is_pre_election: true
I20240503 08:12:44.397471 4436 raft_consensus.cc:2454] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 0.
W20240503 08:12:44.397917 4326 leader_election.cc:341] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:44.398118 4326 leader_election.cc:302] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 3523cd896bb842d68fbb6c15cdd1c3ed; no voters:
I20240503 08:12:44.398732 4481 tablet_bootstrap.cc:492] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35: No bootstrap required, opened a new log
I20240503 08:12:44.398941 4481 ts_tablet_manager.cc:1367] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20240503 08:12:44.399734 4494 raft_consensus.cc:504] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.399772 4481 raft_consensus.cc:348] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.400079 4481 raft_consensus.cc:374] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.400210 4481 raft_consensus.cc:729] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.400362 4494 leader_election.cc:288] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.400547 4494 raft_consensus.cc:2781] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.400636 4494 raft_consensus.cc:482] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.400738 4494 raft_consensus.cc:3037] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.400663 4481 consensus_queue.cc:260] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.401224 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440f42da8514e618147c69fa2795415" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35" is_pre_election: true
I20240503 08:12:44.401566 4481 ts_tablet_manager.cc:1398] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 08:12:44.402518 4251 leader_election.cc:341] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:44.396366 4482 ts_tablet_manager.cc:1367] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20240503 08:12:44.404091 4482 raft_consensus.cc:348] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.404302 4482 raft_consensus.cc:374] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.404377 4482 raft_consensus.cc:729] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.404618 4482 consensus_queue.cc:260] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.405527 4482 ts_tablet_manager.cc:1398] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:44.406036 4494 raft_consensus.cc:504] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.406301 4287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd99f1526db74ef5ace0a312d7afe495" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:44.406498 4287 raft_consensus.cc:3037] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.406672 4494 leader_election.cc:288] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.406872 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440f42da8514e618147c69fa2795415" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" is_pre_election: true
I20240503 08:12:44.407034 4436 raft_consensus.cc:2454] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 0.
I20240503 08:12:44.407285 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7037feb51534b57b0ac70344cdc1e18" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:44.407464 4286 raft_consensus.cc:3037] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.407524 4486 raft_consensus.cc:686] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 LEADER]: Becoming Leader. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:44.407789 4486 consensus_queue.cc:237] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.408140 4488 tablet_bootstrap.cc:492] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35: No bootstrap required, opened a new log
I20240503 08:12:44.408288 4488 ts_tablet_manager.cc:1367] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.022s
I20240503 08:12:44.409008 4488 raft_consensus.cc:348] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.409209 4488 raft_consensus.cc:374] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.409277 4488 raft_consensus.cc:729] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.409409 4287 raft_consensus.cc:2454] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 1.
I20240503 08:12:44.409780 4325 leader_election.cc:302] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.406308 4482 tablet_bootstrap.cc:492] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed: Bootstrap starting.
I20240503 08:12:44.410053 4494 raft_consensus.cc:2781] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.410145 4286 raft_consensus.cc:2454] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 1.
I20240503 08:12:44.410180 4494 raft_consensus.cc:686] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Becoming Leader. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:44.410395 4494 consensus_queue.cc:237] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.410554 4251 leader_election.cc:302] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters: 3482c85dd194402bbc0038963ed16d35
I20240503 08:12:44.410852 4325 leader_election.cc:302] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.411207 4485 raft_consensus.cc:2781] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.411321 4485 raft_consensus.cc:482] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.411448 4485 raft_consensus.cc:3037] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.409507 4488 consensus_queue.cc:260] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.411576 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd99f1526db74ef5ace0a312d7afe495" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:44.412357 4482 tablet_bootstrap.cc:654] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.412353 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7037feb51534b57b0ac70344cdc1e18" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:44.412587 4436 raft_consensus.cc:3037] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.414379 4485 raft_consensus.cc:504] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.415004 4485 leader_election.cc:288] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.415364 4436 raft_consensus.cc:2454] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 1.
I20240503 08:12:44.415886 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440f42da8514e618147c69fa2795415" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:44.416126 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440f42da8514e618147c69fa2795415" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:44.416118 4435 raft_consensus.cc:3037] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.419281 4435 raft_consensus.cc:2454] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 1.
I20240503 08:12:44.419803 4251 leader_election.cc:302] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
W20240503 08:12:44.420176 4251 leader_election.cc:341] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:44.420473 4485 raft_consensus.cc:2781] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.420898 4485 raft_consensus.cc:686] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Leader. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:44.421177 4485 consensus_queue.cc:237] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.424225 4184 catalog_manager.cc:5484] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.426445 4526 raft_consensus.cc:2781] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.426589 4526 raft_consensus.cc:686] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Becoming Leader. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:44.426894 4526 consensus_queue.cc:237] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
W20240503 08:12:44.427824 4326 leader_election.cc:341] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:44.431923 4486 raft_consensus.cc:482] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.431874 4184 catalog_manager.cc:5484] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3). New cstate: current_term: 1 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:44.432082 4486 raft_consensus.cc:504] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.433094 4486 leader_election.cc:288] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:44.433481 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35" is_pre_election: true
I20240503 08:12:44.433560 4485 raft_consensus.cc:482] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.433677 4485 raft_consensus.cc:504] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
W20240503 08:12:44.433900 4400 leader_election.cc:341] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:44.434194 4485 leader_election.cc:288] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.434415 4488 ts_tablet_manager.cc:1398] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35: Time spent starting tablet: real 0.026s user 0.009s sys 0.000s
I20240503 08:12:44.433038 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35" is_pre_election: true
I20240503 08:12:44.434733 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35" is_pre_election: true
I20240503 08:12:44.434821 4286 raft_consensus.cc:2454] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 0.
I20240503 08:12:44.434902 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" is_pre_election: true
I20240503 08:12:44.435112 4436 raft_consensus.cc:2454] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 0.
W20240503 08:12:44.435214 4251 leader_election.cc:341] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:44.435472 4488 tablet_bootstrap.cc:492] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35: Bootstrap starting.
I20240503 08:12:44.435511 4251 leader_election.cc:302] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters: 3482c85dd194402bbc0038963ed16d35
I20240503 08:12:44.435878 4485 raft_consensus.cc:2781] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.435976 4485 raft_consensus.cc:482] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.436065 4485 raft_consensus.cc:3037] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.435415 4399 leader_election.cc:302] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters: 3482c85dd194402bbc0038963ed16d35
I20240503 08:12:44.437258 4488 tablet_bootstrap.cc:654] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.437840 4184 catalog_manager.cc:5484] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3482c85dd194402bbc0038963ed16d35 (127.0.230.2). New cstate: current_term: 1 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.438966 4485 raft_consensus.cc:504] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.440320 4485 leader_election.cc:288] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.440868 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35"
W20240503 08:12:44.441475 4251 leader_election.cc:341] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:44.441893 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:44.442075 4436 raft_consensus.cc:3037] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.444370 4184 catalog_manager.cc:5484] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3482c85dd194402bbc0038963ed16d35 (127.0.230.2). New cstate: current_term: 1 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.445093 4436 raft_consensus.cc:2454] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 1.
I20240503 08:12:44.445533 4251 leader_election.cc:302] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters: 3482c85dd194402bbc0038963ed16d35
I20240503 08:12:44.445955 4485 raft_consensus.cc:2781] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.446373 4485 raft_consensus.cc:686] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Leader. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:44.446656 4485 consensus_queue.cc:237] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.447350 4486 raft_consensus.cc:2741] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 08:12:44.451223 4482 tablet_bootstrap.cc:492] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed: No bootstrap required, opened a new log
I20240503 08:12:44.451427 4482 ts_tablet_manager.cc:1367] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.027s
I20240503 08:12:44.451750 4488 tablet_bootstrap.cc:492] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35: No bootstrap required, opened a new log
I20240503 08:12:44.451581 4184 catalog_manager.cc:5484] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.451879 4488 ts_tablet_manager.cc:1367] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.006s
I20240503 08:12:44.452172 4482 raft_consensus.cc:348] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.452373 4482 raft_consensus.cc:374] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.452462 4482 raft_consensus.cc:729] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.452522 4488 raft_consensus.cc:348] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.452687 4488 raft_consensus.cc:374] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.452777 4488 raft_consensus.cc:729] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.452664 4482 consensus_queue.cc:260] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.452992 4488 consensus_queue.cc:260] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.453182 4482 ts_tablet_manager.cc:1398] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:44.453464 4488 ts_tablet_manager.cc:1398] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:44.453727 4488 tablet_bootstrap.cc:492] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35: Bootstrap starting.
I20240503 08:12:44.453449 4482 tablet_bootstrap.cc:492] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed: Bootstrap starting.
I20240503 08:12:44.455663 4482 tablet_bootstrap.cc:654] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.455664 4488 tablet_bootstrap.cc:654] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.457510 4488 tablet_bootstrap.cc:492] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35: No bootstrap required, opened a new log
I20240503 08:12:44.457660 4488 ts_tablet_manager.cc:1367] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:44.458200 4488 raft_consensus.cc:348] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.458364 4488 raft_consensus.cc:374] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.458439 4488 raft_consensus.cc:729] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.458637 4488 consensus_queue.cc:260] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.458813 4482 tablet_bootstrap.cc:492] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed: No bootstrap required, opened a new log
I20240503 08:12:44.458961 4482 ts_tablet_manager.cc:1367] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240503 08:12:44.459054 4488 ts_tablet_manager.cc:1398] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:44.459266 4488 tablet_bootstrap.cc:492] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35: Bootstrap starting.
I20240503 08:12:44.459638 4482 raft_consensus.cc:348] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.459838 4482 raft_consensus.cc:374] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.459942 4482 raft_consensus.cc:729] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.460165 4482 consensus_queue.cc:260] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.460703 4482 ts_tablet_manager.cc:1398] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:44.460753 4488 tablet_bootstrap.cc:654] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.460984 4482 tablet_bootstrap.cc:492] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed: Bootstrap starting.
I20240503 08:12:44.462482 4482 tablet_bootstrap.cc:654] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.465058 4488 tablet_bootstrap.cc:492] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35: No bootstrap required, opened a new log
I20240503 08:12:44.465178 4488 ts_tablet_manager.cc:1367] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 08:12:44.465663 4488 raft_consensus.cc:348] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.465816 4488 raft_consensus.cc:374] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.465888 4488 raft_consensus.cc:729] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.466069 4488 consensus_queue.cc:260] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.466553 4488 ts_tablet_manager.cc:1398] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:12:44.466818 4488 tablet_bootstrap.cc:492] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35: Bootstrap starting.
I20240503 08:12:44.468343 4488 tablet_bootstrap.cc:654] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.468374 4482 tablet_bootstrap.cc:492] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed: No bootstrap required, opened a new log
I20240503 08:12:44.468499 4482 ts_tablet_manager.cc:1367] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20240503 08:12:44.469048 4482 raft_consensus.cc:348] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.470258 4488 tablet_bootstrap.cc:492] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35: No bootstrap required, opened a new log
I20240503 08:12:44.470355 4488 ts_tablet_manager.cc:1367] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:12:44.470834 4488 raft_consensus.cc:348] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.470961 4488 raft_consensus.cc:374] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.471027 4488 raft_consensus.cc:729] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.471187 4488 consensus_queue.cc:260] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.471594 4488 ts_tablet_manager.cc:1398] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:12:44.472199 4485 raft_consensus.cc:482] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.472317 4485 raft_consensus.cc:504] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.472960 4485 leader_election.cc:288] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.473413 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59aedfe83feb48d799de8eda2afb1860" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35" is_pre_election: true
I20240503 08:12:44.473517 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59aedfe83feb48d799de8eda2afb1860" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" is_pre_election: true
I20240503 08:12:44.473610 4362 raft_consensus.cc:2454] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 0.
I20240503 08:12:44.473665 4436 raft_consensus.cc:2454] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 0.
I20240503 08:12:44.473945 4251 leader_election.cc:302] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.474152 4485 raft_consensus.cc:2781] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.474207 4485 raft_consensus.cc:482] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.474251 4485 raft_consensus.cc:3037] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.474819 4482 raft_consensus.cc:374] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.474882 4482 raft_consensus.cc:729] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.475054 4482 consensus_queue.cc:260] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.475553 4482 ts_tablet_manager.cc:1398] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20240503 08:12:44.476519 4485 raft_consensus.cc:504] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.477005 4485 leader_election.cc:288] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.477432 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59aedfe83feb48d799de8eda2afb1860" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:44.477437 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59aedfe83feb48d799de8eda2afb1860" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:44.477587 4436 raft_consensus.cc:3037] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.477634 4362 raft_consensus.cc:3037] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.479321 4436 raft_consensus.cc:2454] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 1.
I20240503 08:12:44.479658 4251 leader_election.cc:302] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.479774 4362 raft_consensus.cc:2454] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 1.
I20240503 08:12:44.479890 4485 raft_consensus.cc:2781] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.480005 4485 raft_consensus.cc:686] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Leader. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:44.480201 4485 consensus_queue.cc:237] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.482249 4185 catalog_manager.cc:5484] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.488914 4485 raft_consensus.cc:482] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.489014 4485 raft_consensus.cc:504] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.489420 4485 leader_election.cc:288] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.489753 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374e45fb23af484aa3d9c6da1f1863d3" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35" is_pre_election: true
I20240503 08:12:44.489817 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374e45fb23af484aa3d9c6da1f1863d3" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" is_pre_election: true
I20240503 08:12:44.489889 4362 raft_consensus.cc:2454] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 0.
I20240503 08:12:44.489975 4436 raft_consensus.cc:2454] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 0.
I20240503 08:12:44.490182 4251 leader_election.cc:302] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.490396 4485 raft_consensus.cc:2781] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.490463 4485 raft_consensus.cc:482] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.490517 4485 raft_consensus.cc:3037] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.492199 4485 raft_consensus.cc:504] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.492635 4485 leader_election.cc:288] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.492975 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374e45fb23af484aa3d9c6da1f1863d3" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:44.493003 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374e45fb23af484aa3d9c6da1f1863d3" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:44.493098 4362 raft_consensus.cc:3037] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.493108 4436 raft_consensus.cc:3037] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.494771 4362 raft_consensus.cc:2454] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 1.
I20240503 08:12:44.494758 4436 raft_consensus.cc:2454] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 1.
I20240503 08:12:44.495100 4251 leader_election.cc:302] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.495318 4485 raft_consensus.cc:2781] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.495460 4485 raft_consensus.cc:686] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Leader. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:44.495659 4485 consensus_queue.cc:237] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.497362 4526 raft_consensus.cc:482] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.497476 4526 raft_consensus.cc:504] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.498010 4526 leader_election.cc:288] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.497854 4185 catalog_manager.cc:5484] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.498322 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94415b8c11d4f9790344c93eff4c331" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35" is_pre_election: true
I20240503 08:12:44.498351 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94415b8c11d4f9790344c93eff4c331" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" is_pre_election: true
I20240503 08:12:44.498481 4286 raft_consensus.cc:2454] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 0.
I20240503 08:12:44.498499 4436 raft_consensus.cc:2454] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 0.
I20240503 08:12:44.498865 4326 leader_election.cc:302] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.499099 4526 raft_consensus.cc:2781] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.499169 4526 raft_consensus.cc:482] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.499231 4526 raft_consensus.cc:3037] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.501032 4526 raft_consensus.cc:504] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.501463 4526 leader_election.cc:288] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.501755 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94415b8c11d4f9790344c93eff4c331" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:44.501811 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94415b8c11d4f9790344c93eff4c331" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:44.501873 4286 raft_consensus.cc:3037] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.501965 4436 raft_consensus.cc:3037] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.503544 4286 raft_consensus.cc:2454] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 1.
I20240503 08:12:44.503597 4436 raft_consensus.cc:2454] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 1.
I20240503 08:12:44.503861 4325 leader_election.cc:302] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.504082 4526 raft_consensus.cc:2781] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.504190 4526 raft_consensus.cc:686] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Becoming Leader. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:44.504382 4526 consensus_queue.cc:237] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.506350 4185 catalog_manager.cc:5484] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3482c85dd194402bbc0038963ed16d35 (127.0.230.2). New cstate: current_term: 1 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.520449 4485 raft_consensus.cc:482] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.520601 4485 raft_consensus.cc:504] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.521534 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35" is_pre_election: true
I20240503 08:12:44.521657 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" is_pre_election: true
I20240503 08:12:44.521173 4485 leader_election.cc:288] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.521844 4436 raft_consensus.cc:2454] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 0.
I20240503 08:12:44.521711 4362 raft_consensus.cc:2454] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 0.
I20240503 08:12:44.522279 4251 leader_election.cc:302] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.522512 4485 raft_consensus.cc:2781] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.522584 4485 raft_consensus.cc:482] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.522640 4485 raft_consensus.cc:3037] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.525063 4485 raft_consensus.cc:504] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.525475 4485 leader_election.cc:288] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.525868 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:44.525918 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:44.526005 4362 raft_consensus.cc:3037] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.526026 4436 raft_consensus.cc:3037] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.527720 4436 raft_consensus.cc:2454] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 1.
I20240503 08:12:44.527771 4362 raft_consensus.cc:2454] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 1.
I20240503 08:12:44.527985 4251 leader_election.cc:302] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.528170 4485 raft_consensus.cc:2781] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.528270 4485 raft_consensus.cc:686] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Leader. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:44.528460 4485 consensus_queue.cc:237] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.530586 4185 catalog_manager.cc:5484] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.574770 4486 raft_consensus.cc:482] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.574965 4486 raft_consensus.cc:504] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.575624 4486 leader_election.cc:288] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:44.576095 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0dc42483a64c90a1492a7353e2ba4f" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35" is_pre_election: true
I20240503 08:12:44.576097 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0dc42483a64c90a1492a7353e2ba4f" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35" is_pre_election: true
I20240503 08:12:44.576267 4286 raft_consensus.cc:2454] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 0.
I20240503 08:12:44.576318 4362 raft_consensus.cc:2454] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 0.
I20240503 08:12:44.576613 4399 leader_election.cc:302] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.576861 4486 raft_consensus.cc:2781] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.576925 4486 raft_consensus.cc:482] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.576995 4486 raft_consensus.cc:3037] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.578784 4486 raft_consensus.cc:504] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.579272 4486 leader_election.cc:288] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:44.579679 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0dc42483a64c90a1492a7353e2ba4f" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:44.579680 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0dc42483a64c90a1492a7353e2ba4f" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:44.579824 4362 raft_consensus.cc:3037] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.579867 4286 raft_consensus.cc:3037] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.581605 4362 raft_consensus.cc:2454] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 1.
I20240503 08:12:44.581630 4286 raft_consensus.cc:2454] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 1.
I20240503 08:12:44.581919 4399 leader_election.cc:302] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.582131 4486 raft_consensus.cc:2781] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.582266 4486 raft_consensus.cc:686] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 LEADER]: Becoming Leader. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:44.582468 4486 consensus_queue.cc:237] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.584636 4185 catalog_manager.cc:5484] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3). New cstate: current_term: 1 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:44.586234 4486 raft_consensus.cc:482] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.586355 4486 raft_consensus.cc:504] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.586827 4486 leader_election.cc:288] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:44.587173 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd99f1526db74ef5ace0a312d7afe495" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35" is_pre_election: true
I20240503 08:12:44.587273 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd99f1526db74ef5ace0a312d7afe495" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35" is_pre_election: true
I20240503 08:12:44.587440 4286 raft_consensus.cc:2379] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3523cd896bb842d68fbb6c15cdd1c3ed in current term 1: Already voted for candidate 3482c85dd194402bbc0038963ed16d35 in this term.
I20240503 08:12:44.587688 4399 leader_election.cc:302] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed; no voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35
I20240503 08:12:44.587882 4486 raft_consensus.cc:3037] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.589537 4486 raft_consensus.cc:2726] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:44.601907 4485 raft_consensus.cc:482] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.602033 4485 raft_consensus.cc:504] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.602536 4485 leader_election.cc:288] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.602996 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ed061642594372a70df1d766d7b646" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" is_pre_election: true
I20240503 08:12:44.602996 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ed061642594372a70df1d766d7b646" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35" is_pre_election: true
I20240503 08:12:44.603201 4436 raft_consensus.cc:2454] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 0.
I20240503 08:12:44.603201 4362 raft_consensus.cc:2454] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 0.
I20240503 08:12:44.603555 4251 leader_election.cc:302] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.603756 4485 raft_consensus.cc:2781] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.603829 4485 raft_consensus.cc:482] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.603891 4485 raft_consensus.cc:3037] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.605561 4485 raft_consensus.cc:504] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.606037 4485 leader_election.cc:288] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.606400 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ed061642594372a70df1d766d7b646" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:44.606489 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ed061642594372a70df1d766d7b646" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:44.606554 4362 raft_consensus.cc:3037] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.606659 4436 raft_consensus.cc:3037] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.608469 4362 raft_consensus.cc:2454] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 1.
I20240503 08:12:44.608453 4436 raft_consensus.cc:2454] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 1.
I20240503 08:12:44.608762 4251 leader_election.cc:302] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.608942 4485 raft_consensus.cc:2781] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.609055 4485 raft_consensus.cc:686] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Leader. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:44.609244 4485 consensus_queue.cc:237] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.611469 4185 catalog_manager.cc:5484] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.693086 4485 raft_consensus.cc:482] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.693290 4485 raft_consensus.cc:504] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.693902 4485 leader_election.cc:288] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.694387 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35" is_pre_election: true
I20240503 08:12:44.694393 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" is_pre_election: true
I20240503 08:12:44.694564 4362 raft_consensus.cc:2454] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 0.
I20240503 08:12:44.694591 4436 raft_consensus.cc:2454] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 0.
I20240503 08:12:44.694931 4251 leader_election.cc:302] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.695149 4485 raft_consensus.cc:2781] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.695220 4485 raft_consensus.cc:482] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.695271 4485 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.697079 4485 raft_consensus.cc:504] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.697511 4485 leader_election.cc:288] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.697872 4362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:44.697885 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:44.698004 4362 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.698029 4436 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.699887 4362 raft_consensus.cc:2454] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 1.
I20240503 08:12:44.700032 4436 raft_consensus.cc:2454] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 1.
I20240503 08:12:44.700241 4251 leader_election.cc:302] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:44.700453 4485 raft_consensus.cc:2781] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.700568 4485 raft_consensus.cc:686] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Leader. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:44.700794 4485 consensus_queue.cc:237] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.702936 4185 catalog_manager.cc:5484] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1). New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.764276 4362 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.765123 4486 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.766455 4286 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.768926 4486 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.775420 4436 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.776042 4526 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.779630 4287 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.779789 4285 raft_consensus.cc:1261] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.780414 4494 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.780722 4523 consensus_queue.cc:1035] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.784409 4285 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.785077 4531 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.786213 4361 raft_consensus.cc:1261] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.786995 4435 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.787528 4434 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.787695 4285 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:44.788066 4526 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.788908 4486 consensus_queue.cc:1035] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.789988 4494 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.790518 4494 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.811280 4531 raft_consensus.cc:482] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.813915 4531 raft_consensus.cc:504] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.814671 4531 leader_election.cc:288] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.814998 4281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440f42da8514e618147c69fa2795415" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35" is_pre_election: true
I20240503 08:12:44.818081 4433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440f42da8514e618147c69fa2795415" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" is_pre_election: true
I20240503 08:12:44.818435 4433 raft_consensus.cc:2379] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3482c85dd194402bbc0038963ed16d35 in current term 1: Already voted for candidate 8df53dcdf65b407480e6d32372b48a35 in this term.
I20240503 08:12:44.818899 4326 leader_election.cc:302] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35; no voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35
I20240503 08:12:44.819290 4562 mvcc.cc:205] Tried to move back new op lower bound from 7023509359723900928 to 7023509358611886080. Current Snapshot: MvccSnapshot[applied={T|T < 7023509359723900928}]
W20240503 08:12:44.820376 4465 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.0.230.62:44605, user_credentials={real_user=slave}} blocked reactor thread for 53151.1us
I20240503 08:12:44.827814 4361 raft_consensus.cc:1261] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.828948 4485 consensus_queue.cc:1035] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.829598 4531 raft_consensus.cc:3037] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.829649 4433 raft_consensus.cc:1261] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.831323 4582 raft_consensus.cc:482] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.832419 4531 raft_consensus.cc:2726] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:44.831562 4582 raft_consensus.cc:504] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:44.845948 4433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" is_pre_election: true
I20240503 08:12:44.846210 4433 raft_consensus.cc:2379] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3482c85dd194402bbc0038963ed16d35 in current term 1: Already voted for candidate 8df53dcdf65b407480e6d32372b48a35 in this term.
I20240503 08:12:44.845376 4582 leader_election.cc:288] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:44.854867 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df53dcdf65b407480e6d32372b48a35" is_pre_election: true
I20240503 08:12:44.856019 4325 leader_election.cc:302] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35; no voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35
I20240503 08:12:44.856441 4531 raft_consensus.cc:3037] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.859933 4531 raft_consensus.cc:2726] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:44.920318 4238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240503 08:12:44.937685 4251 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.230.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 109432us
I20240503 08:12:44.938373 4436 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.938418 4362 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.939509 4434 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.939920 4362 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.940449 4436 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.941387 4436 raft_consensus.cc:1261] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.941439 4362 raft_consensus.cc:1261] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.942835 4436 raft_consensus.cc:1261] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.944000 4362 raft_consensus.cc:1261] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.944864 4362 raft_consensus.cc:1261] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.944983 4436 raft_consensus.cc:1261] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.945472 4362 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.945416 4485 consensus_queue.cc:1035] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.946578 4593 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.946871 4485 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.947890 4485 consensus_queue.cc:1035] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.947847 4591 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.948637 4593 consensus_queue.cc:1035] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.948900 4485 consensus_queue.cc:1035] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.949592 4284 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1440f42da8514e618147c69fa2795415"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
mode: GRACEFUL
new_leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
from {username='slave'} at 127.0.0.1:57228
I20240503 08:12:44.949785 4284 raft_consensus.cc:595] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Received request to transfer leadership to TS 3523cd896bb842d68fbb6c15cdd1c3ed
I20240503 08:12:44.952549 4285 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
mode: GRACEFUL
new_leader_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:57228
I20240503 08:12:44.952895 4285 raft_consensus.cc:595] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Received request to transfer leadership to TS 3482c85dd194402bbc0038963ed16d35
I20240503 08:12:44.953905 4285 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
mode: GRACEFUL
new_leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
from {username='slave'} at 127.0.0.1:57228
I20240503 08:12:44.954109 4285 raft_consensus.cc:595] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Received request to transfer leadership to TS 3523cd896bb842d68fbb6c15cdd1c3ed
I20240503 08:12:44.954567 4238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 08:12:44.954840 4238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:44.961012 4485 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 08:12:44.962181 4491 consensus_queue.cc:1035] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 08:12:44.962209 4485 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.962458 4591 consensus_queue.cc:1035] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 08:12:44.961478 4592 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240503 08:12:44.964392 4593 consensus_queue.cc:1035] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240503 08:12:44.992415 4593 raft_consensus.cc:979] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35: : Instructing follower 3523cd896bb842d68fbb6c15cdd1c3ed to start an election
I20240503 08:12:45.001377 4593 raft_consensus.cc:1067] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Signalling peer 3523cd896bb842d68fbb6c15cdd1c3ed to start an election
I20240503 08:12:45.002301 4435 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1440f42da8514e618147c69fa2795415"
dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
from {username='slave'} at 127.0.0.1:34898
I20240503 08:12:45.002547 4435 raft_consensus.cc:482] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:45.002683 4435 raft_consensus.cc:3037] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:45.009233 4435 raft_consensus.cc:504] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:45.011312 4285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440f42da8514e618147c69fa2795415" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:45.011570 4285 raft_consensus.cc:3032] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:45.011665 4285 raft_consensus.cc:729] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:45.011965 4285 consensus_queue.cc:260] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:45.012226 4285 raft_consensus.cc:3037] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:45.016146 4285 raft_consensus.cc:2454] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 2.
I20240503 08:12:45.016674 4399 leader_election.cc:302] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:45.017006 4486 raft_consensus.cc:2781] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:45.017143 4486 raft_consensus.cc:686] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Becoming Leader. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:45.017410 4486 consensus_queue.cc:237] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:45.019078 4435 leader_election.cc:288] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 2 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:45.019662 4356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440f42da8514e618147c69fa2795415" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:45.080430 4593 raft_consensus.cc:979] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35: : Instructing follower 3523cd896bb842d68fbb6c15cdd1c3ed to start an election
I20240503 08:12:45.080647 4593 raft_consensus.cc:1067] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Signalling peer 3523cd896bb842d68fbb6c15cdd1c3ed to start an election
I20240503 08:12:45.083576 4434 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4"
dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
from {username='slave'} at 127.0.0.1:34898
I20240503 08:12:45.087042 4434 raft_consensus.cc:482] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:45.091085 4434 raft_consensus.cc:3037] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:45.135324 4592 raft_consensus.cc:979] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35: : Instructing follower 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:45.135193 4566 mvcc.cc:205] Tried to move back new op lower bound from 7023509360073506816 to 7023509358512832512. Current Snapshot: MvccSnapshot[applied={T|T < 7023509360024080384}]
I20240503 08:12:45.136364 4491 raft_consensus.cc:1067] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Signalling peer 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:45.138808 4360 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:50608
I20240503 08:12:45.139035 4360 raft_consensus.cc:482] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:45.139439 4360 raft_consensus.cc:3037] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:45.148694 4434 raft_consensus.cc:504] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:45.149785 4360 raft_consensus.cc:504] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:45.152069 4434 leader_election.cc:288] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 2 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:45.153721 4430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:45.153944 4430 raft_consensus.cc:3037] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:45.153918 4285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:45.155486 4285 raft_consensus.cc:3032] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:45.160364 4284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:45.160579 4284 raft_consensus.cc:3032] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:45.161998 4285 raft_consensus.cc:729] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:45.164372 4360 leader_election.cc:288] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 2 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:45.168378 4284 raft_consensus.cc:729] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
W20240503 08:12:45.174742 4260 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.181114 4267 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
I20240503 08:12:45.189029 4358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
W20240503 08:12:45.191285 4265 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
I20240503 08:12:45.184934 4284 consensus_queue.cc:260] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:45.194772 4358 raft_consensus.cc:3037] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:45.199040 4430 raft_consensus.cc:2454] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 2.
I20240503 08:12:45.203756 4284 raft_consensus.cc:3037] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:12:45.204905 4262 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
I20240503 08:12:45.194299 4285 consensus_queue.cc:260] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
W20240503 08:12:45.208860 4265 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.210638 4260 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
I20240503 08:12:45.214183 4326 leader_election.cc:302] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 3523cd896bb842d68fbb6c15cdd1c3ed; no voters:
I20240503 08:12:45.214432 4285 raft_consensus.cc:3037] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:12:45.216431 4258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.216678 4261 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
I20240503 08:12:45.221462 4529 raft_consensus.cc:2781] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 08:12:45.223790 4261 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.286669 4261 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.298591 4260 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.306121 4265 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
I20240503 08:12:45.312429 4529 raft_consensus.cc:686] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Becoming Leader. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
W20240503 08:12:45.315801 4251 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.230.2:44611, user_credentials={real_user=slave}} blocked reactor thread for 55557.8us
W20240503 08:12:45.317198 4339 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Illegal state: replica 3482c85dd194402bbc0038963ed16d35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.320250 4342 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Service unavailable: leader is not yet ready
W20240503 08:12:45.320676 4341 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Illegal state: replica 3482c85dd194402bbc0038963ed16d35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.321049 4338 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Service unavailable: leader is not yet ready
W20240503 08:12:45.321180 4340 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Service unavailable: leader is not yet ready
I20240503 08:12:45.323951 4529 consensus_queue.cc:237] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
W20240503 08:12:45.324431 4339 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Illegal state: replica 3482c85dd194402bbc0038963ed16d35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.328935 4335 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Illegal state: replica 3482c85dd194402bbc0038963ed16d35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.329233 4337 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Illegal state: replica 3482c85dd194402bbc0038963ed16d35 is not leader of this config: current role FOLLOWER
I20240503 08:12:45.329564 4284 raft_consensus.cc:2454] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 2.
I20240503 08:12:45.324749 4181 catalog_manager.cc:5484] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: term changed from 1 to 2, leader changed from 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) to 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3). New cstate: current_term: 2 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:45.330811 4282 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240503 08:12:45.330852 4333 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Illegal state: replica 3482c85dd194402bbc0038963ed16d35 is not leader of this config: current role FOLLOWER
I20240503 08:12:45.331486 4399 leader_election.cc:302] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
W20240503 08:12:45.339550 4336 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Illegal state: replica 3482c85dd194402bbc0038963ed16d35 is not leader of this config: current role FOLLOWER
I20240503 08:12:45.340490 4359 raft_consensus.cc:3037] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:12:45.343297 4334 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Illegal state: replica 3482c85dd194402bbc0038963ed16d35 is not leader of this config: current role FOLLOWER
I20240503 08:12:45.343501 4527 raft_consensus.cc:2781] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Leader election won for term 2
W20240503 08:12:45.343609 4413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34902: Illegal state: replica 3523cd896bb842d68fbb6c15cdd1c3ed is not leader of this config: current role FOLLOWER
I20240503 08:12:45.343660 4527 raft_consensus.cc:686] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Becoming Leader. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:45.343684 4586 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20240503 08:12:45.344038 4358 raft_consensus.cc:2454] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 2.
I20240503 08:12:45.344004 4527 consensus_queue.cc:237] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:45.346690 4434 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:45.349750 4285 raft_consensus.cc:2454] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 2.
I20240503 08:12:45.353364 4359 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:45.354030 4486 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.355861 4282 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:45.357146 4529 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20240503 08:12:45.358680 4358 raft_consensus.cc:1261] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:45.362391 4527 consensus_queue.cc:1035] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:45.372376 4284 raft_consensus.cc:1261] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:45.374156 4616 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
W20240503 08:12:45.377399 4260 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
I20240503 08:12:45.379992 4486 consensus_queue.cc:1035] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240503 08:12:45.386684 4336 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Illegal state: replica 3482c85dd194402bbc0038963ed16d35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.391229 4340 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50612: Illegal state: replica 3482c85dd194402bbc0038963ed16d35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.397872 4260 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.425601 4260 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.454250 4260 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
W20240503 08:12:45.467118 4261 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57214: Illegal state: replica 8df53dcdf65b407480e6d32372b48a35 is not leader of this config: current role FOLLOWER
I20240503 08:12:45.518826 4179 catalog_manager.cc:5484] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 reported cstate change: term changed from 1 to 2, leader changed from 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) to 3482c85dd194402bbc0038963ed16d35 (127.0.230.2). New cstate: current_term: 2 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:45.526129 4179 catalog_manager.cc:5484] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: term changed from 1 to 2, leader changed from 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) to 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3). New cstate: current_term: 2 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:45.898491 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:45.901157 4632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:45.901643 4633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:45.902074 920 server_base.cc:1034] running on GCE node
W20240503 08:12:45.902407 4635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:45.902702 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:45.902760 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:45.902791 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723965902787 us; error 0 us; skew 500 ppm
I20240503 08:12:45.902912 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:45.904222 920 webserver.cc:460] Webserver started at http://127.0.230.4:42125/ using document root <none> and password file <none>
I20240503 08:12:45.904458 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:45.904522 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:45.904616 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:45.905192 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-3-root/instance:
uuid: "23c12ef68ccc4fad80eb918f77be34e2"
format_stamp: "Formatted at 2024-05-03 08:12:45 on dist-test-slave-2jf9"
I20240503 08:12:45.907689 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:45.909168 4640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:45.909507 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:45.909592 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-3-root
uuid: "23c12ef68ccc4fad80eb918f77be34e2"
format_stamp: "Formatted at 2024-05-03 08:12:45 on dist-test-slave-2jf9"
I20240503 08:12:45.909693 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:45.927726 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:45.928282 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:45.928771 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:45.929591 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:45.929649 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:45.929714 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:45.929757 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:45.943682 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.4:32895
I20240503 08:12:45.943706 4702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.4:32895 every 8 connection(s)
I20240503 08:12:45.945365 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:45.947934 4707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:45.948443 4708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:45.948879 4710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:45.949862 920 server_base.cc:1034] running on GCE node
I20240503 08:12:45.950074 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:45.950124 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:45.950152 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723965950149 us; error 0 us; skew 500 ppm
I20240503 08:12:45.950284 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:45.951120 4703 heartbeater.cc:347] Connected to a master server at 127.0.230.62:44605
I20240503 08:12:45.951234 4703 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:45.951607 4703 heartbeater.cc:511] Master 127.0.230.62:44605 requested a full tablet report, sending...
I20240503 08:12:45.951975 920 webserver.cc:460] Webserver started at http://127.0.230.5:38933/ using document root <none> and password file <none>
I20240503 08:12:45.952145 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:45.952190 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:45.952288 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:45.952440 4181 ts_manager.cc:194] Registered new tserver with Master: 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4:32895)
I20240503 08:12:45.952875 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-4-root/instance:
uuid: "9cb2634fe3314e619359e41fbb351140"
format_stamp: "Formatted at 2024-05-03 08:12:45 on dist-test-slave-2jf9"
I20240503 08:12:45.953148 4181 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56844
I20240503 08:12:45.955418 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:45.956821 4715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:45.957106 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:45.957186 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-4-root
uuid: "9cb2634fe3314e619359e41fbb351140"
format_stamp: "Formatted at 2024-05-03 08:12:45 on dist-test-slave-2jf9"
I20240503 08:12:45.957273 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:45.973887 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:45.974401 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:45.974825 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:45.975627 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:45.975682 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:45.975744 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:45.975777 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:45.987711 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.5:40387
I20240503 08:12:45.987752 4777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.5:40387 every 8 connection(s)
I20240503 08:12:45.989349 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:45.992478 4781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:45.993618 4784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:45.994081 4782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:45.994206 920 server_base.cc:1034] running on GCE node
I20240503 08:12:45.994653 920 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:45.994722 920 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:45.994755 920 hybrid_clock.cc:648] HybridClock initialized: now 1714723965994751 us; error 0 us; skew 500 ppm
I20240503 08:12:45.994910 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:45.995491 4778 heartbeater.cc:347] Connected to a master server at 127.0.230.62:44605
I20240503 08:12:45.995669 4778 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:45.996156 4778 heartbeater.cc:511] Master 127.0.230.62:44605 requested a full tablet report, sending...
I20240503 08:12:45.996366 920 webserver.cc:460] Webserver started at http://127.0.230.6:39475/ using document root <none> and password file <none>
I20240503 08:12:45.996595 920 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:45.996671 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:45.996793 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:45.997042 4181 ts_manager.cc:194] Registered new tserver with Master: 9cb2634fe3314e619359e41fbb351140 (127.0.230.5:40387)
I20240503 08:12:45.997375 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-5-root/instance:
uuid: "981e135f98fb47a1afc7fda11da0e834"
format_stamp: "Formatted at 2024-05-03 08:12:45 on dist-test-slave-2jf9"
I20240503 08:12:45.997789 4181 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56850
I20240503 08:12:45.999918 920 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:46.001252 4789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:46.001525 920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:46.001603 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-5-root
uuid: "981e135f98fb47a1afc7fda11da0e834"
format_stamp: "Formatted at 2024-05-03 08:12:45 on dist-test-slave-2jf9"
I20240503 08:12:46.001689 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:46.023070 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:46.023622 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:46.024086 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:46.024850 920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:46.024909 920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:46.024978 920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:46.025014 920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:46.037416 920 rpc_server.cc:307] RPC server started. Bound to: 127.0.230.6:39275
I20240503 08:12:46.037453 4851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.230.6:39275 every 8 connection(s)
I20240503 08:12:46.043838 4852 heartbeater.cc:347] Connected to a master server at 127.0.230.62:44605
I20240503 08:12:46.043960 4852 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:46.044374 4852 heartbeater.cc:511] Master 127.0.230.62:44605 requested a full tablet report, sending...
I20240503 08:12:46.045148 4181 ts_manager.cc:194] Registered new tserver with Master: 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6:39275)
I20240503 08:12:46.045887 4181 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56856
I20240503 08:12:46.947597 4281 consensus_queue.cc:237] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:46.949946 4361 raft_consensus.cc:1261] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:12:46.949940 4432 raft_consensus.cc:1261] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:12:46.950490 4859 consensus_queue.cc:1035] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:46.950942 4858 consensus_queue.cc:1035] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:46.953444 4593 raft_consensus.cc:2932] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:46.954416 4703 heartbeater.cc:503] Master 127.0.230.62:44605 was elected leader, sending a full tablet report...
I20240503 08:12:46.955209 4238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:46.955478 4432 raft_consensus.cc:2932] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:46.955509 4361 raft_consensus.cc:2932] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:46.958060 4238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:46.958231 4238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:46.959273 4184 catalog_manager.cc:5484] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: config changed from index -1 to 27, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:46.960268 4250 consensus_peers.cc:485] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 -> Peer 9cb2634fe3314e619359e41fbb351140 (127.0.230.5:40387): Couldn't send request to peer 9cb2634fe3314e619359e41fbb351140. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 374e45fb23af484aa3d9c6da1f1863d3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:46.960656 4281 consensus_queue.cc:237] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:12:46.963089 4361 raft_consensus.cc:1261] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:12:46.963176 4432 raft_consensus.cc:1261] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:12:46.963606 4858 consensus_queue.cc:1035] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:12:46.963891 4593 consensus_queue.cc:1035] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
W20240503 08:12:46.970887 4252 consensus_peers.cc:485] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 -> Peer 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4:32895): Couldn't send request to peer 23c12ef68ccc4fad80eb918f77be34e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59aedfe83feb48d799de8eda2afb1860. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:46.978464 4593 raft_consensus.cc:2932] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:12:46.979951 4361 raft_consensus.cc:2932] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:12:46.980422 4432 raft_consensus.cc:2932] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:12:46.986050 4179 catalog_manager.cc:5484] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: config changed from index -1 to 26, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:12:46.996845 4361 consensus_queue.cc:237] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:12:46.999013 4778 heartbeater.cc:503] Master 127.0.230.62:44605 was elected leader, sending a full tablet report...
I20240503 08:12:46.999873 4281 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:12:47.000244 4432 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:12:47.000370 4494 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:47.000775 4875 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:47.002946 4494 raft_consensus.cc:2932] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:12:47.003890 4432 raft_consensus.cc:2932] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:12:47.006031 4283 raft_consensus.cc:2932] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
W20240503 08:12:47.007761 4326 consensus_peers.cc:485] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 -> Peer 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6:39275): Couldn't send request to peer 981e135f98fb47a1afc7fda11da0e834. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd99f1526db74ef5ace0a312d7afe495. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:47.008353 4181 catalog_manager.cc:5484] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: config changed from index -1 to 27, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New cstate: current_term: 1 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:12:47.010676 4361 consensus_queue.cc:237] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:12:47.013567 4283 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 08:12:47.013674 4432 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 08:12:47.014139 4881 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:47.015157 4881 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:47.016724 4494 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:12:47.018127 4283 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
W20240503 08:12:47.019520 4327 consensus_peers.cc:485] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 -> Peer 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4:32895): Couldn't send request to peer 23c12ef68ccc4fad80eb918f77be34e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2b69fcfaa7742919bc05385b49623dd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:47.021014 4184 catalog_manager.cc:5484] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 reported cstate change: config changed from index -1 to 28, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New cstate: current_term: 2 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:47.022567 4432 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:12:47.028674 4432 consensus_queue.cc:237] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:47.030988 4361 raft_consensus.cc:1261] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:12:47.031455 4283 raft_consensus.cc:1261] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:12:47.031603 4631 consensus_queue.cc:1035] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:12:47.034220 4873 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:47.039000 4361 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:47.039778 4355 consensus_queue.cc:237] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:12:47.042402 4432 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20240503 08:12:47.042418 4326 consensus_peers.cc:485] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 -> Peer 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6:39275): Couldn't send request to peer 981e135f98fb47a1afc7fda11da0e834. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c94415b8c11d4f9790344c93eff4c331. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:47.042857 4875 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:47.043606 4184 catalog_manager.cc:5484] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 reported cstate change: config changed from index -1 to 26, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New cstate: current_term: 1 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:47.044591 4887 consensus_queue.cc:1035] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:12:47.045552 4283 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:12:47.046123 4880 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:47.046937 4852 heartbeater.cc:503] Master 127.0.230.62:44605 was elected leader, sending a full tablet report...
W20240503 08:12:47.056591 4399 consensus_peers.cc:485] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed -> Peer 9cb2634fe3314e619359e41fbb351140 (127.0.230.5:40387): Couldn't send request to peer 9cb2634fe3314e619359e41fbb351140. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d0dc42483a64c90a1492a7353e2ba4f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:47.058415 4281 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:47.059125 4880 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:12:47.060928 4432 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:12:47.064750 4184 catalog_manager.cc:5484] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: config changed from index -1 to 27, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New cstate: current_term: 1 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:47.066102 4286 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:12:47.407478 4894 ts_tablet_manager.cc:902] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2: Initiating tablet copy from peer 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947)
I20240503 08:12:47.409124 4894 tablet_copy_client.cc:308] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.1:32947
I20240503 08:12:47.414911 4297 tablet_copy_service.cc:140] P 8df53dcdf65b407480e6d32372b48a35: Received BeginTabletCopySession request for tablet 59aedfe83feb48d799de8eda2afb1860 from peer 23c12ef68ccc4fad80eb918f77be34e2 ({username='slave'} at 127.0.0.1:57252)
I20240503 08:12:47.415083 4297 tablet_copy_service.cc:161] P 8df53dcdf65b407480e6d32372b48a35: Beginning new tablet copy session on tablet 59aedfe83feb48d799de8eda2afb1860 from peer 23c12ef68ccc4fad80eb918f77be34e2 at {username='slave'} at 127.0.0.1:57252: session id = 23c12ef68ccc4fad80eb918f77be34e2-59aedfe83feb48d799de8eda2afb1860
I20240503 08:12:47.417238 4297 tablet_copy_source_session.cc:205] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:47.418493 4894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59aedfe83feb48d799de8eda2afb1860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.423739 4894 tablet_copy_client.cc:783] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:47.424000 4894 tablet_copy_client.cc:647] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:47.425053 4900 ts_tablet_manager.cc:902] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140: Initiating tablet copy from peer 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947)
I20240503 08:12:47.425559 4898 tablet_copy_client.cc:1002] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:47.427186 4900 tablet_copy_client.cc:308] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.1:32947
I20240503 08:12:47.432894 4297 tablet_copy_service.cc:140] P 8df53dcdf65b407480e6d32372b48a35: Received BeginTabletCopySession request for tablet 374e45fb23af484aa3d9c6da1f1863d3 from peer 9cb2634fe3314e619359e41fbb351140 ({username='slave'} at 127.0.0.1:57262)
I20240503 08:12:47.433033 4297 tablet_copy_service.cc:161] P 8df53dcdf65b407480e6d32372b48a35: Beginning new tablet copy session on tablet 374e45fb23af484aa3d9c6da1f1863d3 from peer 9cb2634fe3314e619359e41fbb351140 at {username='slave'} at 127.0.0.1:57262: session id = 9cb2634fe3314e619359e41fbb351140-374e45fb23af484aa3d9c6da1f1863d3
I20240503 08:12:47.434757 4297 tablet_copy_source_session.cc:205] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:47.435808 4900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374e45fb23af484aa3d9c6da1f1863d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.440789 4900 tablet_copy_client.cc:783] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:47.441071 4900 tablet_copy_client.cc:647] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:47.442490 4904 tablet_copy_client.cc:1002] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:47.477321 4906 ts_tablet_manager.cc:902] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: Initiating tablet copy from peer 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:47.479249 4906 tablet_copy_client.cc:308] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.2:44611
I20240503 08:12:47.484326 4911 ts_tablet_manager.cc:902] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834: Initiating tablet copy from peer 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:47.484486 4372 tablet_copy_service.cc:140] P 3482c85dd194402bbc0038963ed16d35: Received BeginTabletCopySession request for tablet b2b69fcfaa7742919bc05385b49623dd from peer 23c12ef68ccc4fad80eb918f77be34e2 ({username='slave'} at 127.0.0.1:50628)
I20240503 08:12:47.484634 4372 tablet_copy_service.cc:161] P 3482c85dd194402bbc0038963ed16d35: Beginning new tablet copy session on tablet b2b69fcfaa7742919bc05385b49623dd from peer 23c12ef68ccc4fad80eb918f77be34e2 at {username='slave'} at 127.0.0.1:50628: session id = 23c12ef68ccc4fad80eb918f77be34e2-b2b69fcfaa7742919bc05385b49623dd
I20240503 08:12:47.485561 4911 tablet_copy_client.cc:308] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.2:44611
I20240503 08:12:47.487171 4372 tablet_copy_source_session.cc:205] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:47.488454 4906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b69fcfaa7742919bc05385b49623dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.492146 4372 tablet_copy_service.cc:140] P 3482c85dd194402bbc0038963ed16d35: Received BeginTabletCopySession request for tablet c94415b8c11d4f9790344c93eff4c331 from peer 981e135f98fb47a1afc7fda11da0e834 ({username='slave'} at 127.0.0.1:50634)
I20240503 08:12:47.492384 4372 tablet_copy_service.cc:161] P 3482c85dd194402bbc0038963ed16d35: Beginning new tablet copy session on tablet c94415b8c11d4f9790344c93eff4c331 from peer 981e135f98fb47a1afc7fda11da0e834 at {username='slave'} at 127.0.0.1:50634: session id = 981e135f98fb47a1afc7fda11da0e834-c94415b8c11d4f9790344c93eff4c331
I20240503 08:12:47.493649 4906 tablet_copy_client.cc:783] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:47.493964 4906 tablet_copy_client.cc:647] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:47.495476 4910 tablet_copy_client.cc:1002] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:47.495539 4372 tablet_copy_source_session.cc:205] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:47.496469 4911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94415b8c11d4f9790344c93eff4c331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.501154 4911 tablet_copy_client.cc:783] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:47.501386 4911 tablet_copy_client.cc:647] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:47.502780 4915 tablet_copy_client.cc:1002] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:47.528329 4917 ts_tablet_manager.cc:902] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834: Initiating tablet copy from peer 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:47.529603 4917 tablet_copy_client.cc:308] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.2:44611
I20240503 08:12:47.530174 4372 tablet_copy_service.cc:140] P 3482c85dd194402bbc0038963ed16d35: Received BeginTabletCopySession request for tablet bd99f1526db74ef5ace0a312d7afe495 from peer 981e135f98fb47a1afc7fda11da0e834 ({username='slave'} at 127.0.0.1:50634)
I20240503 08:12:47.530330 4372 tablet_copy_service.cc:161] P 3482c85dd194402bbc0038963ed16d35: Beginning new tablet copy session on tablet bd99f1526db74ef5ace0a312d7afe495 from peer 981e135f98fb47a1afc7fda11da0e834 at {username='slave'} at 127.0.0.1:50634: session id = 981e135f98fb47a1afc7fda11da0e834-bd99f1526db74ef5ace0a312d7afe495
I20240503 08:12:47.533115 4372 tablet_copy_source_session.cc:205] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:47.534045 4917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd99f1526db74ef5ace0a312d7afe495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.538152 4917 tablet_copy_client.cc:783] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:47.538340 4917 tablet_copy_client.cc:647] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:47.539543 4921 tablet_copy_client.cc:1002] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:47.550530 4922 ts_tablet_manager.cc:902] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140: Initiating tablet copy from peer 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:47.552167 4922 tablet_copy_client.cc:308] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.3:38567
I20240503 08:12:47.557752 4446 tablet_copy_service.cc:140] P 3523cd896bb842d68fbb6c15cdd1c3ed: Received BeginTabletCopySession request for tablet 2d0dc42483a64c90a1492a7353e2ba4f from peer 9cb2634fe3314e619359e41fbb351140 ({username='slave'} at 127.0.0.1:34910)
I20240503 08:12:47.557888 4446 tablet_copy_service.cc:161] P 3523cd896bb842d68fbb6c15cdd1c3ed: Beginning new tablet copy session on tablet 2d0dc42483a64c90a1492a7353e2ba4f from peer 9cb2634fe3314e619359e41fbb351140 at {username='slave'} at 127.0.0.1:34910: session id = 9cb2634fe3314e619359e41fbb351140-2d0dc42483a64c90a1492a7353e2ba4f
I20240503 08:12:47.560187 4446 tablet_copy_source_session.cc:205] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:47.561311 4922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d0dc42483a64c90a1492a7353e2ba4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.566932 4922 tablet_copy_client.cc:783] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:47.567231 4922 tablet_copy_client.cc:647] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:47.568761 4926 tablet_copy_client.cc:1002] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:48.958585 4238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:48.961321 4238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:12:48.961585 4238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:49.426505 4894 tablet_copy_client.cc:523] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:49.429873 4894 tablet_bootstrap.cc:492] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2: Bootstrap starting.
I20240503 08:12:49.443614 4900 tablet_copy_client.cc:523] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:49.446414 4900 tablet_bootstrap.cc:492] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140: Bootstrap starting.
I20240503 08:12:49.467460 4894 tablet_bootstrap.cc:492] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=102 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:49.467824 4894 tablet_bootstrap.cc:492] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2: Bootstrap complete.
I20240503 08:12:49.468088 4894 ts_tablet_manager.cc:1367] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2: Time spent bootstrapping tablet: real 0.038s user 0.034s sys 0.005s
I20240503 08:12:49.468833 4894 raft_consensus.cc:348] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:12:49.469043 4894 raft_consensus.cc:729] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 23c12ef68ccc4fad80eb918f77be34e2, State: Initialized, Role: LEARNER
I20240503 08:12:49.469228 4894 consensus_queue.cc:260] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:12:49.469966 4894 ts_tablet_manager.cc:1398] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:49.470669 4297 tablet_copy_service.cc:342] P 8df53dcdf65b407480e6d32372b48a35: Request end of tablet copy session 23c12ef68ccc4fad80eb918f77be34e2-59aedfe83feb48d799de8eda2afb1860 received from {username='slave'} at 127.0.0.1:57252
I20240503 08:12:49.470778 4297 tablet_copy_service.cc:433] P 8df53dcdf65b407480e6d32372b48a35: ending tablet copy session 23c12ef68ccc4fad80eb918f77be34e2-59aedfe83feb48d799de8eda2afb1860 on tablet 59aedfe83feb48d799de8eda2afb1860 with peer 23c12ef68ccc4fad80eb918f77be34e2
I20240503 08:12:49.482179 4900 tablet_bootstrap.cc:492] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=92 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:49.482635 4900 tablet_bootstrap.cc:492] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140: Bootstrap complete.
I20240503 08:12:49.482887 4900 ts_tablet_manager.cc:1367] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140: Time spent bootstrapping tablet: real 0.037s user 0.034s sys 0.000s
I20240503 08:12:49.483554 4900 raft_consensus.cc:348] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:49.483899 4900 raft_consensus.cc:729] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9cb2634fe3314e619359e41fbb351140, State: Initialized, Role: LEARNER
I20240503 08:12:49.484123 4900 consensus_queue.cc:260] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:49.484891 4900 ts_tablet_manager.cc:1398] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20240503 08:12:49.485491 4297 tablet_copy_service.cc:342] P 8df53dcdf65b407480e6d32372b48a35: Request end of tablet copy session 9cb2634fe3314e619359e41fbb351140-374e45fb23af484aa3d9c6da1f1863d3 received from {username='slave'} at 127.0.0.1:57262
I20240503 08:12:49.485605 4297 tablet_copy_service.cc:433] P 8df53dcdf65b407480e6d32372b48a35: ending tablet copy session 9cb2634fe3314e619359e41fbb351140-374e45fb23af484aa3d9c6da1f1863d3 on tablet 374e45fb23af484aa3d9c6da1f1863d3 with peer 9cb2634fe3314e619359e41fbb351140
I20240503 08:12:49.496069 4906 tablet_copy_client.cc:523] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:49.499162 4906 tablet_bootstrap.cc:492] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: Bootstrap starting.
I20240503 08:12:49.503276 4911 tablet_copy_client.cc:523] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:49.506574 4911 tablet_bootstrap.cc:492] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834: Bootstrap starting.
I20240503 08:12:49.539594 4906 tablet_bootstrap.cc:492] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=107 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:49.540010 4917 tablet_copy_client.cc:523] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:49.543511 4917 tablet_bootstrap.cc:492] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834: Bootstrap starting.
I20240503 08:12:49.546001 4911 tablet_bootstrap.cc:492] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=107 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:49.546398 4911 tablet_bootstrap.cc:492] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834: Bootstrap complete.
I20240503 08:12:49.546655 4911 ts_tablet_manager.cc:1367] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834: Time spent bootstrapping tablet: real 0.040s user 0.035s sys 0.004s
I20240503 08:12:49.547448 4911 raft_consensus.cc:348] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:12:49.547761 4911 raft_consensus.cc:729] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 981e135f98fb47a1afc7fda11da0e834, State: Initialized, Role: LEARNER
I20240503 08:12:49.548043 4911 consensus_queue.cc:260] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:12:49.549057 4911 ts_tablet_manager.cc:1398] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 08:12:49.550088 4372 tablet_copy_service.cc:342] P 3482c85dd194402bbc0038963ed16d35: Request end of tablet copy session 981e135f98fb47a1afc7fda11da0e834-c94415b8c11d4f9790344c93eff4c331 received from {username='slave'} at 127.0.0.1:50634
I20240503 08:12:49.550213 4372 tablet_copy_service.cc:433] P 3482c85dd194402bbc0038963ed16d35: ending tablet copy session 981e135f98fb47a1afc7fda11da0e834-c94415b8c11d4f9790344c93eff4c331 on tablet c94415b8c11d4f9790344c93eff4c331 with peer 981e135f98fb47a1afc7fda11da0e834
I20240503 08:12:49.552417 4906 tablet_bootstrap.cc:492] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: Bootstrap complete.
I20240503 08:12:49.552723 4906 ts_tablet_manager.cc:1367] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: Time spent bootstrapping tablet: real 0.054s user 0.038s sys 0.004s
I20240503 08:12:49.553361 4906 raft_consensus.cc:348] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:12:49.553575 4906 raft_consensus.cc:729] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 23c12ef68ccc4fad80eb918f77be34e2, State: Initialized, Role: LEARNER
I20240503 08:12:49.553799 4906 consensus_queue.cc:260] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:12:49.554800 4906 ts_tablet_manager.cc:1398] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:49.555366 4372 tablet_copy_service.cc:342] P 3482c85dd194402bbc0038963ed16d35: Request end of tablet copy session 23c12ef68ccc4fad80eb918f77be34e2-b2b69fcfaa7742919bc05385b49623dd received from {username='slave'} at 127.0.0.1:50628
I20240503 08:12:49.555493 4372 tablet_copy_service.cc:433] P 3482c85dd194402bbc0038963ed16d35: ending tablet copy session 23c12ef68ccc4fad80eb918f77be34e2-b2b69fcfaa7742919bc05385b49623dd on tablet b2b69fcfaa7742919bc05385b49623dd with peer 23c12ef68ccc4fad80eb918f77be34e2
I20240503 08:12:49.559295 4678 raft_consensus.cc:1203] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20240503 08:12:49.569375 4922 tablet_copy_client.cc:523] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:49.572111 4922 tablet_bootstrap.cc:492] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140: Bootstrap starting.
I20240503 08:12:49.589258 4917 tablet_bootstrap.cc:492] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=116 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:49.589784 4917 tablet_bootstrap.cc:492] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834: Bootstrap complete.
I20240503 08:12:49.590095 4917 ts_tablet_manager.cc:1367] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834: Time spent bootstrapping tablet: real 0.047s user 0.039s sys 0.000s
I20240503 08:12:49.590880 4917 raft_consensus.cc:348] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:12:49.591168 4917 raft_consensus.cc:729] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 981e135f98fb47a1afc7fda11da0e834, State: Initialized, Role: LEARNER
I20240503 08:12:49.591425 4917 consensus_queue.cc:260] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:12:49.592007 4917 ts_tablet_manager.cc:1398] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:49.592615 4372 tablet_copy_service.cc:342] P 3482c85dd194402bbc0038963ed16d35: Request end of tablet copy session 981e135f98fb47a1afc7fda11da0e834-bd99f1526db74ef5ace0a312d7afe495 received from {username='slave'} at 127.0.0.1:50634
I20240503 08:12:49.592723 4372 tablet_copy_service.cc:433] P 3482c85dd194402bbc0038963ed16d35: ending tablet copy session 981e135f98fb47a1afc7fda11da0e834-bd99f1526db74ef5ace0a312d7afe495 on tablet bd99f1526db74ef5ace0a312d7afe495 with peer 981e135f98fb47a1afc7fda11da0e834
I20240503 08:12:49.608430 4922 tablet_bootstrap.cc:492] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:49.608780 4922 tablet_bootstrap.cc:492] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140: Bootstrap complete.
I20240503 08:12:49.608996 4922 ts_tablet_manager.cc:1367] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140: Time spent bootstrapping tablet: real 0.037s user 0.029s sys 0.003s
I20240503 08:12:49.609632 4922 raft_consensus.cc:348] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:49.609813 4922 raft_consensus.cc:729] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9cb2634fe3314e619359e41fbb351140, State: Initialized, Role: LEARNER
I20240503 08:12:49.609987 4922 consensus_queue.cc:260] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:49.610370 4922 ts_tablet_manager.cc:1398] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:12:49.610942 4446 tablet_copy_service.cc:342] P 3523cd896bb842d68fbb6c15cdd1c3ed: Request end of tablet copy session 9cb2634fe3314e619359e41fbb351140-2d0dc42483a64c90a1492a7353e2ba4f received from {username='slave'} at 127.0.0.1:34910
I20240503 08:12:49.611059 4446 tablet_copy_service.cc:433] P 3523cd896bb842d68fbb6c15cdd1c3ed: ending tablet copy session 9cb2634fe3314e619359e41fbb351140-2d0dc42483a64c90a1492a7353e2ba4f on tablet 2d0dc42483a64c90a1492a7353e2ba4f with peer 9cb2634fe3314e619359e41fbb351140
I20240503 08:12:49.686760 4753 raft_consensus.cc:1203] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240503 08:12:49.713443 4827 raft_consensus.cc:1203] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 08:12:49.782441 4678 raft_consensus.cc:1203] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240503 08:12:49.887686 4753 raft_consensus.cc:1203] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 08:12:49.932354 4827 raft_consensus.cc:1203] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 08:12:50.133412 4946 raft_consensus.cc:1050] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed: attempting to promote NON_VOTER 9cb2634fe3314e619359e41fbb351140 to VOTER
I20240503 08:12:50.134171 4946 consensus_queue.cc:237] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:50.136426 4355 raft_consensus.cc:1261] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:12:50.136493 4286 raft_consensus.cc:1261] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:12:50.136497 4753 raft_consensus.cc:1261] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140 [term 1 LEARNER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:12:50.136849 4934 consensus_queue.cc:1035] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:50.137091 4946 consensus_queue.cc:1035] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:50.137564 4945 consensus_queue.cc:1035] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:50.140484 4946 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.141003 4753 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.141603 4286 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.145087 4361 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.145964 4179 catalog_manager.cc:5484] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: config changed from index 26 to 27, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.156935 4432 consensus_queue.cc:237] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:50.158643 4753 raft_consensus.cc:1261] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.158645 4286 raft_consensus.cc:1261] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.159067 4945 consensus_queue.cc:1035] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.159336 4946 consensus_queue.cc:1035] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.161219 4934 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.164717 4286 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.169029 4170 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d0dc42483a64c90a1492a7353e2ba4f with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:50.169585 4182 catalog_manager.cc:5484] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: config changed from index 27 to 28, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New cstate: current_term: 1 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.169775 4753 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.176203 4875 raft_consensus.cc:1050] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35: attempting to promote NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 to VOTER
I20240503 08:12:50.176668 4351 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d0dc42483a64c90a1492a7353e2ba4f with delete_type TABLET_DATA_TOMBSTONED (TS 3482c85dd194402bbc0038963ed16d35 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:50578
I20240503 08:12:50.176961 4875 consensus_queue.cc:237] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:50.177650 4957 tablet_replica.cc:330] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35: stopping tablet replica
I20240503 08:12:50.178231 4957 raft_consensus.cc:2229] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:50.178433 4957 raft_consensus.cc:2258] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.179493 4957 ts_tablet_manager.cc:1875] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:50.185034 4286 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:12:50.185184 4432 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:12:50.185585 4956 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:50.186125 4955 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:50.186429 4678 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 LEARNER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:12:50.186952 4957 ts_tablet_manager.cc:1888] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240503 08:12:50.187178 4957 log.cc:1196] T 2d0dc42483a64c90a1492a7353e2ba4f P 3482c85dd194402bbc0038963ed16d35: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-1-root/wals/2d0dc42483a64c90a1492a7353e2ba4f
I20240503 08:12:50.188117 4170 catalog_manager.cc:4830] TS 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611): tablet 2d0dc42483a64c90a1492a7353e2ba4f (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:50.188431 4955 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:50.192309 4286 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.192533 4959 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.196358 4181 catalog_manager.cc:5484] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: config changed from index 28 to 29, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.198050 4432 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.204512 4678 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.207217 4361 consensus_queue.cc:237] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:50.209219 4281 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:12:50.209234 4678 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:12:50.209748 4956 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:50.210088 4875 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:50.212146 4959 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.212504 4286 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.212625 4678 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.215305 4170 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b2b69fcfaa7742919bc05385b49623dd with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:50.216758 4181 catalog_manager.cc:5484] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 reported cstate change: config changed from index 29 to 30, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New cstate: current_term: 2 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:50.219921 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b69fcfaa7742919bc05385b49623dd with delete_type TABLET_DATA_TOMBSTONED (TS 3523cd896bb842d68fbb6c15cdd1c3ed not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:34878
I20240503 08:12:50.223795 4963 tablet_replica.cc:330] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed: stopping tablet replica
I20240503 08:12:50.223951 4963 raft_consensus.cc:2229] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:50.224128 4963 raft_consensus.cc:2258] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.224808 4963 ts_tablet_manager.cc:1875] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:50.229763 4963 ts_tablet_manager.cc:1888] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240503 08:12:50.229841 4963 log.cc:1196] T b2b69fcfaa7742919bc05385b49623dd P 3523cd896bb842d68fbb6c15cdd1c3ed: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-2-root/wals/b2b69fcfaa7742919bc05385b49623dd
I20240503 08:12:50.230455 4170 catalog_manager.cc:4830] TS 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567): tablet b2b69fcfaa7742919bc05385b49623dd (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:50.281066 4965 raft_consensus.cc:1050] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35: attempting to promote NON_VOTER 9cb2634fe3314e619359e41fbb351140 to VOTER
I20240503 08:12:50.281833 4965 consensus_queue.cc:237] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:50.283934 4753 raft_consensus.cc:1261] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [term 1 LEARNER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.284320 4432 raft_consensus.cc:1261] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.284415 4858 consensus_queue.cc:1035] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.284508 4361 raft_consensus.cc:1261] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.285107 4964 consensus_queue.cc:1035] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.285925 4858 consensus_queue.cc:1035] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.292846 4964 raft_consensus.cc:2932] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.293720 4753 raft_consensus.cc:2932] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.298955 4184 catalog_manager.cc:5484] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 reported cstate change: config changed from index 27 to 28, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.299489 4875 raft_consensus.cc:1050] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35: attempting to promote NON_VOTER 981e135f98fb47a1afc7fda11da0e834 to VOTER
I20240503 08:12:50.299300 4361 raft_consensus.cc:2932] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.300292 4875 consensus_queue.cc:237] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } }
I20240503 08:12:50.304774 4827 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 1 LEARNER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.304793 4430 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.305558 4281 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.305469 4956 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.305989 4956 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.306604 4956 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.309875 4430 raft_consensus.cc:2932] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.312026 4955 raft_consensus.cc:2932] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.312382 4286 consensus_queue.cc:237] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:50.314227 4827 raft_consensus.cc:2932] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.315600 4434 raft_consensus.cc:2932] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.316939 4753 raft_consensus.cc:1261] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:12:50.316855 4184 catalog_manager.cc:5484] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: config changed from index 27 to 28, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:50.317378 4858 consensus_queue.cc:1035] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:50.320118 4964 raft_consensus.cc:2932] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.322274 4432 raft_consensus.cc:1261] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:12:50.323048 4965 consensus_queue.cc:1035] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:50.323274 4169 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 374e45fb23af484aa3d9c6da1f1863d3 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:50.324052 4752 raft_consensus.cc:2932] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.324694 4432 raft_consensus.cc:2932] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.328541 4355 consensus_queue.cc:237] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } }
I20240503 08:12:50.329428 4181 catalog_manager.cc:5484] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: config changed from index 28 to 29, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:50.330245 4827 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:12:50.330644 4875 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:50.332825 4432 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:12:50.333745 4351 tablet_service.cc:1514] Processing DeleteTablet for tablet 374e45fb23af484aa3d9c6da1f1863d3 with delete_type TABLET_DATA_TOMBSTONED (TS 3482c85dd194402bbc0038963ed16d35 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:50578
I20240503 08:12:50.334022 4957 tablet_replica.cc:330] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35: stopping tablet replica
I20240503 08:12:50.334195 4957 raft_consensus.cc:2229] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:50.334270 4955 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:50.334370 4957 raft_consensus.cc:2258] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.334249 4956 raft_consensus.cc:2932] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.335318 4957 ts_tablet_manager.cc:1875] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:50.335177 4826 raft_consensus.cc:2932] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.337527 4432 raft_consensus.cc:2932] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.340124 4170 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bd99f1526db74ef5ace0a312d7afe495 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:50.340577 4182 catalog_manager.cc:5484] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 reported cstate change: config changed from index 28 to 29, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New cstate: current_term: 1 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.340837 4964 raft_consensus.cc:1050] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35: attempting to promote NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 to VOTER
I20240503 08:12:50.341682 4964 consensus_queue.cc:237] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:50.344156 4273 tablet_service.cc:1514] Processing DeleteTablet for tablet bd99f1526db74ef5ace0a312d7afe495 with delete_type TABLET_DATA_TOMBSTONED (TS 8df53dcdf65b407480e6d32372b48a35 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:57198
I20240503 08:12:50.344856 4976 tablet_replica.cc:330] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35: stopping tablet replica
I20240503 08:12:50.345012 4957 ts_tablet_manager.cc:1888] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 08:12:50.345036 4976 raft_consensus.cc:2229] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:50.345106 4957 log.cc:1196] T 374e45fb23af484aa3d9c6da1f1863d3 P 3482c85dd194402bbc0038963ed16d35: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-1-root/wals/374e45fb23af484aa3d9c6da1f1863d3
I20240503 08:12:50.345165 4976 pending_rounds.cc:62] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35: Trying to abort 1 pending ops.
I20240503 08:12:50.345273 4976 pending_rounds.cc:68] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35: Aborting op as it isn't in flight: id { term: 1 index: 28 } timestamp: 7023509382347788288 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "bd99f1526db74ef5ace0a312d7afe495" old_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } } new_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } } }
I20240503 08:12:50.345715 4170 catalog_manager.cc:4830] TS 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611): tablet 374e45fb23af484aa3d9c6da1f1863d3 (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:50.345749 4976 raft_consensus.cc:2894] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Aborting config change with OpId 1.28: Aborted: Op aborted
I20240503 08:12:50.345862 4976 raft_consensus.cc:2866] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:12:50.345917 4976 raft_consensus.cc:2258] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.346556 4976 ts_tablet_manager.cc:1875] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:50.348647 4678 raft_consensus.cc:1261] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [term 1 LEARNER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:12:50.349098 4858 consensus_queue.cc:1035] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:50.349627 4361 raft_consensus.cc:1261] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:12:50.350139 4432 raft_consensus.cc:1261] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:12:50.350812 4965 consensus_queue.cc:1035] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:50.351269 4965 consensus_queue.cc:1035] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:50.352075 4976 ts_tablet_manager.cc:1888] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 08:12:50.352231 4976 log.cc:1196] T bd99f1526db74ef5ace0a312d7afe495 P 8df53dcdf65b407480e6d32372b48a35: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-0-root/wals/bd99f1526db74ef5ace0a312d7afe495
I20240503 08:12:50.353219 4169 catalog_manager.cc:4830] TS 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947): tablet bd99f1526db74ef5ace0a312d7afe495 (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:50.356887 4965 raft_consensus.cc:2932] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.359962 4678 raft_consensus.cc:2932] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.361079 4181 catalog_manager.cc:5484] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: config changed from index 26 to 27, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:50.362543 4361 raft_consensus.cc:2932] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.363487 4432 raft_consensus.cc:2932] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.367431 4281 consensus_queue.cc:237] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:50.370623 4355 raft_consensus.cc:1261] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.370870 4678 raft_consensus.cc:1261] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [term 1 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.371279 4858 consensus_queue.cc:1035] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.371646 4858 consensus_queue.cc:1035] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.373804 4965 raft_consensus.cc:2932] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.374377 4678 raft_consensus.cc:2932] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.375622 4361 raft_consensus.cc:2932] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.377708 4169 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 59aedfe83feb48d799de8eda2afb1860 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:50.380004 4184 catalog_manager.cc:5484] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 reported cstate change: config changed from index 27 to 28, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New cstate: current_term: 1 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:50.383572 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet 59aedfe83feb48d799de8eda2afb1860 with delete_type TABLET_DATA_TOMBSTONED (TS 3523cd896bb842d68fbb6c15cdd1c3ed not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:34878
I20240503 08:12:50.383869 4963 tablet_replica.cc:330] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed: stopping tablet replica
I20240503 08:12:50.384045 4963 raft_consensus.cc:2229] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:50.384392 4963 raft_consensus.cc:2258] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.385600 4963 ts_tablet_manager.cc:1875] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:50.389580 4963 ts_tablet_manager.cc:1888] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240503 08:12:50.389667 4963 log.cc:1196] T 59aedfe83feb48d799de8eda2afb1860 P 3523cd896bb842d68fbb6c15cdd1c3ed: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-2-root/wals/59aedfe83feb48d799de8eda2afb1860
I20240503 08:12:50.397547 4170 catalog_manager.cc:4830] TS 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567): tablet 59aedfe83feb48d799de8eda2afb1860 (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:50.413475 4955 raft_consensus.cc:1050] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35: attempting to promote NON_VOTER 981e135f98fb47a1afc7fda11da0e834 to VOTER
I20240503 08:12:50.414148 4955 consensus_queue.cc:237] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } }
I20240503 08:12:50.415930 4826 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 LEARNER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.415989 4432 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.416378 4959 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.416579 4286 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:12:50.416648 4875 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.417147 4959 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:50.419446 4875 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.422359 4826 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.422950 4286 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.423328 4184 catalog_manager.cc:5484] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: config changed from index 27 to 28, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:50.426721 4432 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.431456 4355 consensus_queue.cc:237] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } }
I20240503 08:12:50.433099 4432 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:12:50.433151 4826 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:12:50.434221 4956 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:50.434470 4955 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:50.436296 4956 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.436955 4432 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.439622 4826 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:50.441895 4170 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c94415b8c11d4f9790344c93eff4c331 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:50.442222 4184 catalog_manager.cc:5484] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: config changed from index 28 to 29, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New cstate: current_term: 1 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:50.447535 4273 tablet_service.cc:1514] Processing DeleteTablet for tablet c94415b8c11d4f9790344c93eff4c331 with delete_type TABLET_DATA_TOMBSTONED (TS 8df53dcdf65b407480e6d32372b48a35 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:57198
I20240503 08:12:50.447839 4976 tablet_replica.cc:330] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35: stopping tablet replica
I20240503 08:12:50.448009 4976 raft_consensus.cc:2229] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:50.448155 4976 raft_consensus.cc:2258] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.448766 4976 ts_tablet_manager.cc:1875] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:50.452780 4976 ts_tablet_manager.cc:1888] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 08:12:50.452864 4976 log.cc:1196] T c94415b8c11d4f9790344c93eff4c331 P 8df53dcdf65b407480e6d32372b48a35: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-0-root/wals/c94415b8c11d4f9790344c93eff4c331
I20240503 08:12:50.453362 4169 catalog_manager.cc:4830] TS 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947): tablet c94415b8c11d4f9790344c93eff4c331 (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:50.961952 4238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:50.973368 4432 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240"
dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
mode: GRACEFUL
new_leader_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:34920
I20240503 08:12:50.973579 4432 raft_consensus.cc:595] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 LEADER]: Received request to transfer leadership to TS 3482c85dd194402bbc0038963ed16d35
I20240503 08:12:50.974501 4432 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1440f42da8514e618147c69fa2795415"
dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
mode: GRACEFUL
new_leader_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:34920
I20240503 08:12:50.974653 4432 raft_consensus.cc:595] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Received request to transfer leadership to TS 3482c85dd194402bbc0038963ed16d35
I20240503 08:12:50.975466 4286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "374e45fb23af484aa3d9c6da1f1863d3"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
mode: GRACEFUL
new_leader_uuid: "9cb2634fe3314e619359e41fbb351140"
from {username='slave'} at 127.0.0.1:57228
I20240503 08:12:50.975587 4286 raft_consensus.cc:595] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Received request to transfer leadership to TS 9cb2634fe3314e619359e41fbb351140
I20240503 08:12:50.980749 4355 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
mode: GRACEFUL
new_leader_uuid: "23c12ef68ccc4fad80eb918f77be34e2"
from {username='slave'} at 127.0.0.1:50644
I20240503 08:12:50.980890 4355 raft_consensus.cc:595] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Received request to transfer leadership to TS 23c12ef68ccc4fad80eb918f77be34e2
I20240503 08:12:50.981737 4355 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d7037feb51534b57b0ac70344cdc1e18"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
mode: GRACEFUL
new_leader_uuid: "8df53dcdf65b407480e6d32372b48a35"
from {username='slave'} at 127.0.0.1:50644
I20240503 08:12:50.981864 4355 raft_consensus.cc:595] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Received request to transfer leadership to TS 8df53dcdf65b407480e6d32372b48a35
I20240503 08:12:50.982753 4286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8ed061642594372a70df1d766d7b646"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
mode: GRACEFUL
new_leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
from {username='slave'} at 127.0.0.1:57228
I20240503 08:12:50.982887 4286 raft_consensus.cc:595] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Received request to transfer leadership to TS 3523cd896bb842d68fbb6c15cdd1c3ed
I20240503 08:12:50.983215 4238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240503 08:12:50.983485 4238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:51.015816 4959 raft_consensus.cc:979] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35: : Instructing follower 8df53dcdf65b407480e6d32372b48a35 to start an election
I20240503 08:12:51.015973 4959 raft_consensus.cc:1067] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Signalling peer 8df53dcdf65b407480e6d32372b48a35 to start an election
I20240503 08:12:51.016448 4286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7037feb51534b57b0ac70344cdc1e18"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
from {username='slave'} at 127.0.0.1:57212
I20240503 08:12:51.016599 4286 raft_consensus.cc:482] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:51.016696 4286 raft_consensus.cc:3037] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:51.018503 4286 raft_consensus.cc:504] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.018975 4286 leader_election.cc:288] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 2 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:51.019304 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7037feb51534b57b0ac70344cdc1e18" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:51.019368 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7037feb51534b57b0ac70344cdc1e18" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:51.019483 4355 raft_consensus.cc:3032] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:51.019546 4355 raft_consensus.cc:729] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:51.019556 4432 raft_consensus.cc:3037] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:51.019753 4355 consensus_queue.cc:260] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.020006 4355 raft_consensus.cc:3037] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:51.021296 4432 raft_consensus.cc:2454] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 2.
I20240503 08:12:51.021622 4251 leader_election.cc:302] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:51.021647 4355 raft_consensus.cc:2454] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 2.
I20240503 08:12:51.021845 4985 raft_consensus.cc:2781] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:51.021953 4985 raft_consensus.cc:686] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 2 LEADER]: Becoming Leader. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:51.022163 4985 consensus_queue.cc:237] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.024513 4182 catalog_manager.cc:5484] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: term changed from 1 to 2, leader changed from 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) to 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1). New cstate: current_term: 2 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:51.028762 4858 raft_consensus.cc:979] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35: : Instructing follower 3523cd896bb842d68fbb6c15cdd1c3ed to start an election
I20240503 08:12:51.028873 4858 raft_consensus.cc:1067] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Signalling peer 3523cd896bb842d68fbb6c15cdd1c3ed to start an election
I20240503 08:12:51.029314 4432 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8ed061642594372a70df1d766d7b646"
dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
from {username='slave'} at 127.0.0.1:34898
I20240503 08:12:51.029436 4432 raft_consensus.cc:482] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:51.029525 4432 raft_consensus.cc:3037] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:51.031109 4432 raft_consensus.cc:504] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.031589 4432 leader_election.cc:288] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 2 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:51.031936 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ed061642594372a70df1d766d7b646" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:51.031947 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ed061642594372a70df1d766d7b646" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:51.032065 4286 raft_consensus.cc:3032] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:51.032086 4355 raft_consensus.cc:3037] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:51.032125 4286 raft_consensus.cc:729] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:51.032320 4286 consensus_queue.cc:260] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.032565 4286 raft_consensus.cc:3037] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:51.033708 4355 raft_consensus.cc:2454] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 2.
I20240503 08:12:51.034021 4400 leader_election.cc:302] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 3523cd896bb842d68fbb6c15cdd1c3ed; no voters:
I20240503 08:12:51.034240 4934 raft_consensus.cc:2781] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:51.034310 4286 raft_consensus.cc:2454] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 2.
I20240503 08:12:51.034585 4934 raft_consensus.cc:686] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Becoming Leader. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:51.034816 4934 consensus_queue.cc:237] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.036897 4182 catalog_manager.cc:5484] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: term changed from 1 to 2, leader changed from 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) to 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3). New cstate: current_term: 2 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:51.047796 4993 raft_consensus.cc:979] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed: : Instructing follower 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:51.047923 4993 raft_consensus.cc:1067] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Signalling peer 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:51.048417 4355 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1440f42da8514e618147c69fa2795415"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:50592
I20240503 08:12:51.048554 4355 raft_consensus.cc:482] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:51.048616 4355 raft_consensus.cc:3037] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:51.050197 4355 raft_consensus.cc:504] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.050645 4355 leader_election.cc:288] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 3 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:51.050954 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440f42da8514e618147c69fa2795415" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:51.051021 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1440f42da8514e618147c69fa2795415" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:51.051113 4286 raft_consensus.cc:3037] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:51.051141 4432 raft_consensus.cc:3032] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:12:51.051189 4432 raft_consensus.cc:729] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:51.051381 4432 consensus_queue.cc:260] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.051678 4432 raft_consensus.cc:3037] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:51.053009 4286 raft_consensus.cc:2454] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 3.
I20240503 08:12:51.053333 4325 leader_election.cc:302] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:51.053390 4432 raft_consensus.cc:2454] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 3.
I20240503 08:12:51.053576 4959 raft_consensus.cc:2781] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:12:51.053690 4959 raft_consensus.cc:686] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Becoming Leader. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:51.053892 4959 consensus_queue.cc:237] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.056176 4182 catalog_manager.cc:5484] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: term changed from 2 to 3, leader changed from 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) to 3482c85dd194402bbc0038963ed16d35 (127.0.230.2). New cstate: current_term: 3 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:51.057803 4934 raft_consensus.cc:979] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed: : Instructing follower 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:51.057925 4934 raft_consensus.cc:1067] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 LEADER]: Signalling peer 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:51.058396 4355 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:50592
I20240503 08:12:51.058584 4355 raft_consensus.cc:482] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:51.058661 4355 raft_consensus.cc:3037] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:51.060395 4355 raft_consensus.cc:504] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.060808 4355 leader_election.cc:288] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 2 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:51.061085 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:51.061213 4286 raft_consensus.cc:3037] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:51.061182 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:51.061353 4432 raft_consensus.cc:3032] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:51.061409 4432 raft_consensus.cc:729] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:51.061590 4432 consensus_queue.cc:260] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.061836 4432 raft_consensus.cc:3037] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:51.063169 4286 raft_consensus.cc:2454] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 2.
I20240503 08:12:51.063501 4325 leader_election.cc:302] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:51.063714 4432 raft_consensus.cc:2454] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 2.
I20240503 08:12:51.063712 4959 raft_consensus.cc:2781] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:51.063859 4959 raft_consensus.cc:686] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Becoming Leader. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:51.064064 4959 consensus_queue.cc:237] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.066270 4182 catalog_manager.cc:5484] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: term changed from 1 to 2, leader changed from 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) to 3482c85dd194402bbc0038963ed16d35 (127.0.230.2). New cstate: current_term: 2 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:51.210438 4956 raft_consensus.cc:979] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35: : Instructing follower 23c12ef68ccc4fad80eb918f77be34e2 to start an election
I20240503 08:12:51.210592 4956 raft_consensus.cc:1067] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Signalling peer 23c12ef68ccc4fad80eb918f77be34e2 to start an election
I20240503 08:12:51.211128 4678 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd"
dest_uuid: "23c12ef68ccc4fad80eb918f77be34e2"
from {username='slave'} at 127.0.0.1:51704
I20240503 08:12:51.211320 4678 raft_consensus.cc:482] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:51.211426 4678 raft_consensus.cc:3037] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:51.213305 4678 raft_consensus.cc:504] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:51.213999 4678 leader_election.cc:288] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [CANDIDATE]: Term 3 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:51.214313 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd" candidate_uuid: "23c12ef68ccc4fad80eb918f77be34e2" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:51.214483 4286 raft_consensus.cc:3037] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:51.214488 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b69fcfaa7742919bc05385b49623dd" candidate_uuid: "23c12ef68ccc4fad80eb918f77be34e2" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:51.214679 4355 raft_consensus.cc:3032] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:12:51.214728 4355 raft_consensus.cc:729] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:51.214946 4355 consensus_queue.cc:260] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:51.215313 4355 raft_consensus.cc:3037] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:51.216846 4286 raft_consensus.cc:2454] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c12ef68ccc4fad80eb918f77be34e2 in term 3.
I20240503 08:12:51.217140 4641 leader_election.cc:302] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c12ef68ccc4fad80eb918f77be34e2, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:51.217223 4355 raft_consensus.cc:2454] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c12ef68ccc4fad80eb918f77be34e2 in term 3.
I20240503 08:12:51.217583 4997 raft_consensus.cc:2781] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:12:51.217888 4997 raft_consensus.cc:686] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 3 LEADER]: Becoming Leader. State: Replica: 23c12ef68ccc4fad80eb918f77be34e2, State: Running, Role: LEADER
I20240503 08:12:51.218083 4997 consensus_queue.cc:237] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:51.220443 4182 catalog_manager.cc:5484] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 reported cstate change: term changed from 2 to 3, leader changed from 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) to 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4). New cstate: current_term: 3 leader_uuid: "23c12ef68ccc4fad80eb918f77be34e2" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:51.310865 4985 raft_consensus.cc:979] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35: : Instructing follower 9cb2634fe3314e619359e41fbb351140 to start an election
I20240503 08:12:51.311055 4985 raft_consensus.cc:1067] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Signalling peer 9cb2634fe3314e619359e41fbb351140 to start an election
I20240503 08:12:51.311615 4752 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "374e45fb23af484aa3d9c6da1f1863d3"
dest_uuid: "9cb2634fe3314e619359e41fbb351140"
from {username='slave'} at 127.0.0.1:58436
I20240503 08:12:51.311796 4752 raft_consensus.cc:482] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:51.311887 4752 raft_consensus.cc:3037] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:51.314373 4752 raft_consensus.cc:504] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:51.315016 4752 leader_election.cc:288] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [CANDIDATE]: Term 2 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:51.315415 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374e45fb23af484aa3d9c6da1f1863d3" candidate_uuid: "9cb2634fe3314e619359e41fbb351140" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:51.315410 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374e45fb23af484aa3d9c6da1f1863d3" candidate_uuid: "9cb2634fe3314e619359e41fbb351140" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:51.315604 4286 raft_consensus.cc:3032] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:51.315644 4432 raft_consensus.cc:3037] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:51.315672 4286 raft_consensus.cc:729] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:51.315876 4286 consensus_queue.cc:260] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:51.316124 4286 raft_consensus.cc:3037] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:51.317734 4432 raft_consensus.cc:2454] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb2634fe3314e619359e41fbb351140 in term 2.
I20240503 08:12:51.317929 4286 raft_consensus.cc:2454] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb2634fe3314e619359e41fbb351140 in term 2.
I20240503 08:12:51.318048 4717 leader_election.cc:302] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 9cb2634fe3314e619359e41fbb351140; no voters:
I20240503 08:12:51.318537 5000 raft_consensus.cc:2781] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:51.318836 5000 raft_consensus.cc:686] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [term 2 LEADER]: Becoming Leader. State: Replica: 9cb2634fe3314e619359e41fbb351140, State: Running, Role: LEADER
I20240503 08:12:51.319052 5000 consensus_queue.cc:237] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:51.320931 4182 catalog_manager.cc:5484] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 reported cstate change: term changed from 1 to 2, leader changed from 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) to 9cb2634fe3314e619359e41fbb351140 (127.0.230.5). New cstate: current_term: 2 leader_uuid: "9cb2634fe3314e619359e41fbb351140" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:51.406235 4355 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 08:12:51.406693 4858 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:51.415238 4432 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 08:12:51.415745 5005 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:51.436062 4286 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:12:51.436558 4956 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:51.439699 4432 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:12:51.440315 5009 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:51.448578 4286 consensus_queue.cc:237] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:51.450176 4355 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 08:12:51.450248 4432 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
W20240503 08:12:51.450297 4250 consensus_peers.cc:485] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 -> Peer 9cb2634fe3314e619359e41fbb351140 (127.0.230.5:40387): Couldn't send request to peer 9cb2634fe3314e619359e41fbb351140. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7037feb51534b57b0ac70344cdc1e18. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:51.450578 5005 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:51.450791 4985 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:51.452415 5005 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:51.452890 4355 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:51.453166 4432 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:51.457237 4184 catalog_manager.cc:5484] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: config changed from index -1 to 28, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New cstate: current_term: 2 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:51.457649 4233 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240503 08:12:51.460836 4233 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f8ed061642594372a70df1d766d7b646, TS 3482c85dd194402bbc0038963ed16d35 -> TS 23c12ef68ccc4fad80eb918f77be34e2 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
W20240503 08:12:51.460907 4233 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f8ed061642594372a70df1d766d7b646, TS 3482c85dd194402bbc0038963ed16d35 -> TS 23c12ef68ccc4fad80eb918f77be34e2 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
W20240503 08:12:51.463102 4233 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b82e1e5d27e441ccbd62d1296c06a2f4, TS 8df53dcdf65b407480e6d32372b48a35 -> TS 981e135f98fb47a1afc7fda11da0e834 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
W20240503 08:12:51.463160 4233 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b82e1e5d27e441ccbd62d1296c06a2f4, TS 8df53dcdf65b407480e6d32372b48a35 -> TS 981e135f98fb47a1afc7fda11da0e834 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
W20240503 08:12:51.465440 4233 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 03e50d2db8824a11af7a5e4f9f3ed240, TS 3523cd896bb842d68fbb6c15cdd1c3ed -> TS 9cb2634fe3314e619359e41fbb351140 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
W20240503 08:12:51.465512 4233 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 03e50d2db8824a11af7a5e4f9f3ed240, TS 3523cd896bb842d68fbb6c15cdd1c3ed -> TS 9cb2634fe3314e619359e41fbb351140 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
W20240503 08:12:51.472299 4233 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 374e45fb23af484aa3d9c6da1f1863d3, TS 8df53dcdf65b407480e6d32372b48a35 -> TS 23c12ef68ccc4fad80eb918f77be34e2 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
W20240503 08:12:51.472357 4233 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 374e45fb23af484aa3d9c6da1f1863d3, TS 8df53dcdf65b407480e6d32372b48a35 -> TS 23c12ef68ccc4fad80eb918f77be34e2 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
W20240503 08:12:51.474418 4233 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ae715a227487424dbe08825a7f7d9819, TS 3482c85dd194402bbc0038963ed16d35 -> TS 981e135f98fb47a1afc7fda11da0e834 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
W20240503 08:12:51.474474 4233 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ae715a227487424dbe08825a7f7d9819, TS 3482c85dd194402bbc0038963ed16d35 -> TS 981e135f98fb47a1afc7fda11da0e834 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:51.515208 4355 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 08:12:51.515733 4993 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:51.519781 4286 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 08:12:51.520283 4934 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:51.560216 4430 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 08:12:51.560689 5009 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:51.564251 4281 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 08:12:51.564687 4959 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:51.769611 4355 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Refusing update from remote peer 23c12ef68ccc4fad80eb918f77be34e2: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 08:12:51.770107 5018 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:51.774880 4281 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Refusing update from remote peer 23c12ef68ccc4fad80eb918f77be34e2: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 08:12:51.775470 5018 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:51.799022 4430 raft_consensus.cc:1261] T 374e45fb23af484aa3d9c6da1f1863d3 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Refusing update from remote peer 9cb2634fe3314e619359e41fbb351140: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:12:51.799561 5000 consensus_queue.cc:1035] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:51.804998 4286 raft_consensus.cc:1261] T 374e45fb23af484aa3d9c6da1f1863d3 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Refusing update from remote peer 9cb2634fe3314e619359e41fbb351140: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:12:51.805495 5001 consensus_queue.cc:1035] T 374e45fb23af484aa3d9c6da1f1863d3 P 9cb2634fe3314e619359e41fbb351140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:51.988068 5028 ts_tablet_manager.cc:902] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140: Initiating tablet copy from peer 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947)
I20240503 08:12:51.989404 5028 tablet_copy_client.cc:308] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.1:32947
I20240503 08:12:51.989956 4297 tablet_copy_service.cc:140] P 8df53dcdf65b407480e6d32372b48a35: Received BeginTabletCopySession request for tablet d7037feb51534b57b0ac70344cdc1e18 from peer 9cb2634fe3314e619359e41fbb351140 ({username='slave'} at 127.0.0.1:57262)
I20240503 08:12:51.990101 4297 tablet_copy_service.cc:161] P 8df53dcdf65b407480e6d32372b48a35: Beginning new tablet copy session on tablet d7037feb51534b57b0ac70344cdc1e18 from peer 9cb2634fe3314e619359e41fbb351140 at {username='slave'} at 127.0.0.1:57262: session id = 9cb2634fe3314e619359e41fbb351140-d7037feb51534b57b0ac70344cdc1e18
I20240503 08:12:51.992146 4297 tablet_copy_source_session.cc:205] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:51.993196 5028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7037feb51534b57b0ac70344cdc1e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.998061 5028 tablet_copy_client.cc:783] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:51.998245 5028 tablet_copy_client.cc:647] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:51.999748 5032 tablet_copy_client.cc:1002] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:52.983789 4238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:52.986867 4355 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
mode: GRACEFUL
new_leader_uuid: "8df53dcdf65b407480e6d32372b48a35"
from {username='slave'} at 127.0.0.1:50644
I20240503 08:12:52.987056 4355 raft_consensus.cc:595] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Received request to transfer leadership to TS 8df53dcdf65b407480e6d32372b48a35
I20240503 08:12:52.987993 4281 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
mode: GRACEFUL
new_leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
from {username='slave'} at 127.0.0.1:57228
I20240503 08:12:52.988184 4281 raft_consensus.cc:595] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Received request to transfer leadership to TS 3523cd896bb842d68fbb6c15cdd1c3ed
I20240503 08:12:52.989109 4355 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd99f1526db74ef5ace0a312d7afe495"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
mode: GRACEFUL
new_leader_uuid: "981e135f98fb47a1afc7fda11da0e834"
from {username='slave'} at 127.0.0.1:50644
I20240503 08:12:52.989302 4355 raft_consensus.cc:595] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Received request to transfer leadership to TS 981e135f98fb47a1afc7fda11da0e834
I20240503 08:12:52.990160 4430 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8ed061642594372a70df1d766d7b646"
dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
mode: GRACEFUL
new_leader_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:34920
I20240503 08:12:52.990309 4430 raft_consensus.cc:595] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Received request to transfer leadership to TS 3482c85dd194402bbc0038963ed16d35
I20240503 08:12:52.990649 4238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240503 08:12:52.990872 4238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:53.084594 5034 raft_consensus.cc:979] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed: : Instructing follower 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:53.084786 5034 raft_consensus.cc:1067] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Signalling peer 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:53.085371 4355 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8ed061642594372a70df1d766d7b646"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:50592
I20240503 08:12:53.085534 4355 raft_consensus.cc:482] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:53.085615 4355 raft_consensus.cc:3037] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:53.088315 4355 raft_consensus.cc:504] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:53.089015 4355 leader_election.cc:288] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 3 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:53.089246 4430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ed061642594372a70df1d766d7b646" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:53.089401 4430 raft_consensus.cc:3032] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:12:53.089450 4430 raft_consensus.cc:729] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:53.089623 4430 consensus_queue.cc:260] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:53.089844 4430 raft_consensus.cc:3037] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:53.091585 4430 raft_consensus.cc:2454] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 3.
I20240503 08:12:53.091878 4281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ed061642594372a70df1d766d7b646" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:53.091924 4326 leader_election.cc:302] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 3523cd896bb842d68fbb6c15cdd1c3ed; no voters:
I20240503 08:12:53.092037 4281 raft_consensus.cc:3037] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:53.092154 4956 raft_consensus.cc:2781] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:12:53.092523 4956 raft_consensus.cc:686] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Becoming Leader. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:53.092772 4956 consensus_queue.cc:237] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:53.094595 4281 raft_consensus.cc:2454] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 3.
I20240503 08:12:53.095853 4184 catalog_manager.cc:5484] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: term changed from 2 to 3, leader changed from 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) to 3482c85dd194402bbc0038963ed16d35 (127.0.230.2). New cstate: current_term: 3 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:53.178645 5035 raft_consensus.cc:979] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35: : Instructing follower 3523cd896bb842d68fbb6c15cdd1c3ed to start an election
I20240503 08:12:53.178817 5035 raft_consensus.cc:1067] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Signalling peer 3523cd896bb842d68fbb6c15cdd1c3ed to start an election
I20240503 08:12:53.179323 4430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819"
dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
from {username='slave'} at 127.0.0.1:34898
I20240503 08:12:53.179536 4430 raft_consensus.cc:482] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:53.179636 4430 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:53.181768 4430 raft_consensus.cc:504] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:53.182309 4430 leader_election.cc:288] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 2 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:53.182646 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:53.182639 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:53.182847 4355 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:53.182942 4286 raft_consensus.cc:3032] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:53.182996 4286 raft_consensus.cc:729] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:53.183209 4286 consensus_queue.cc:260] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:53.183493 4286 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:53.184963 4355 raft_consensus.cc:2454] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 2.
I20240503 08:12:53.185178 4286 raft_consensus.cc:2454] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 2.
I20240503 08:12:53.185427 4400 leader_election.cc:302] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 3523cd896bb842d68fbb6c15cdd1c3ed; no voters:
I20240503 08:12:53.185688 5037 raft_consensus.cc:2781] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:53.185828 5037 raft_consensus.cc:686] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Becoming Leader. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:53.186051 5037 consensus_queue.cc:237] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:53.188548 4184 catalog_manager.cc:5484] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: term changed from 1 to 2, leader changed from 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) to 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3). New cstate: current_term: 2 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:53.335682 5039 raft_consensus.cc:979] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35: : Instructing follower 981e135f98fb47a1afc7fda11da0e834 to start an election
I20240503 08:12:53.335850 5039 raft_consensus.cc:1067] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Signalling peer 981e135f98fb47a1afc7fda11da0e834 to start an election
I20240503 08:12:53.336397 4826 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bd99f1526db74ef5ace0a312d7afe495"
dest_uuid: "981e135f98fb47a1afc7fda11da0e834"
from {username='slave'} at 127.0.0.1:56336
I20240503 08:12:53.336612 4826 raft_consensus.cc:482] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:53.336721 4826 raft_consensus.cc:3037] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:53.339179 4826 raft_consensus.cc:504] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } }
I20240503 08:12:53.340335 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd99f1526db74ef5ace0a312d7afe495" candidate_uuid: "981e135f98fb47a1afc7fda11da0e834" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:53.340552 4355 raft_consensus.cc:3032] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:53.340621 4355 raft_consensus.cc:729] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:53.340863 4355 consensus_queue.cc:260] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } }
I20240503 08:12:53.341166 4355 raft_consensus.cc:3037] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:53.341239 4826 leader_election.cc:288] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [CANDIDATE]: Term 2 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:53.343021 4355 raft_consensus.cc:2454] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 981e135f98fb47a1afc7fda11da0e834 in term 2.
I20240503 08:12:53.351634 4791 leader_election.cc:302] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 981e135f98fb47a1afc7fda11da0e834; no voters:
I20240503 08:12:53.352315 5048 raft_consensus.cc:2781] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:53.353087 5048 raft_consensus.cc:686] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 2 LEADER]: Becoming Leader. State: Replica: 981e135f98fb47a1afc7fda11da0e834, State: Running, Role: LEADER
I20240503 08:12:53.353391 5048 consensus_queue.cc:237] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } }
I20240503 08:12:53.356227 4184 catalog_manager.cc:5484] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 reported cstate change: term changed from 1 to 2, leader changed from 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) to 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6). New cstate: current_term: 2 leader_uuid: "981e135f98fb47a1afc7fda11da0e834" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:53.359045 4430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd99f1526db74ef5ace0a312d7afe495" candidate_uuid: "981e135f98fb47a1afc7fda11da0e834" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:53.359265 4430 raft_consensus.cc:3037] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:53.361348 4430 raft_consensus.cc:2454] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 981e135f98fb47a1afc7fda11da0e834 in term 2.
I20240503 08:12:53.480751 4956 raft_consensus.cc:979] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35: : Instructing follower 8df53dcdf65b407480e6d32372b48a35 to start an election
I20240503 08:12:53.480895 4956 raft_consensus.cc:1067] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Signalling peer 8df53dcdf65b407480e6d32372b48a35 to start an election
I20240503 08:12:53.481405 4286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
from {username='slave'} at 127.0.0.1:57212
I20240503 08:12:53.481571 4286 raft_consensus.cc:482] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:53.481666 4286 raft_consensus.cc:3037] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:53.483925 4286 raft_consensus.cc:504] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:53.484400 4286 leader_election.cc:288] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 3 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:53.484778 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:53.484833 4430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:53.484947 4355 raft_consensus.cc:3032] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:12:53.485008 4355 raft_consensus.cc:729] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:53.485034 4430 raft_consensus.cc:3037] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:53.485211 4355 consensus_queue.cc:260] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:53.485503 4355 raft_consensus.cc:3037] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:53.487217 4430 raft_consensus.cc:2454] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 3.
I20240503 08:12:53.487571 4251 leader_election.cc:302] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:53.487617 4355 raft_consensus.cc:2454] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 3.
I20240503 08:12:53.487830 5036 raft_consensus.cc:2781] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:12:53.487943 5036 raft_consensus.cc:686] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 3 LEADER]: Becoming Leader. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:53.488176 5036 consensus_queue.cc:237] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:53.491533 4184 catalog_manager.cc:5484] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: term changed from 2 to 3, leader changed from 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) to 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1). New cstate: current_term: 3 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:53.650599 4286 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:12:53.651091 4956 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:53.655038 4430 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:12:53.655511 5039 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:53.752310 4281 raft_consensus.cc:1261] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 08:12:53.752808 5037 consensus_queue.cc:1035] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:53.757102 4355 raft_consensus.cc:1261] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 08:12:53.758872 5058 consensus_queue.cc:1035] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:12:53.790782 4355 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Refusing update from remote peer 981e135f98fb47a1afc7fda11da0e834: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:12:53.791242 5048 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:53.794842 4430 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Refusing update from remote peer 981e135f98fb47a1afc7fda11da0e834: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:12:53.795589 5049 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:53.867982 4355 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:12:53.868490 5036 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:53.874042 4430 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:12:53.874569 5036 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:54.000059 5028 tablet_copy_client.cc:523] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:54.003242 5028 tablet_bootstrap.cc:492] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140: Bootstrap starting.
I20240503 08:12:54.035838 5028 tablet_bootstrap.cc:492] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=95 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:54.036159 5028 tablet_bootstrap.cc:492] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140: Bootstrap complete.
I20240503 08:12:54.036376 5028 ts_tablet_manager.cc:1367] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140: Time spent bootstrapping tablet: real 0.033s user 0.035s sys 0.000s
I20240503 08:12:54.036971 5028 raft_consensus.cc:348] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:54.037142 5028 raft_consensus.cc:729] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9cb2634fe3314e619359e41fbb351140, State: Initialized, Role: LEARNER
I20240503 08:12:54.037292 5028 consensus_queue.cc:260] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:54.037772 5028 ts_tablet_manager.cc:1398] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:54.038426 4297 tablet_copy_service.cc:342] P 8df53dcdf65b407480e6d32372b48a35: Request end of tablet copy session 9cb2634fe3314e619359e41fbb351140-d7037feb51534b57b0ac70344cdc1e18 received from {username='slave'} at 127.0.0.1:57262
I20240503 08:12:54.038549 4297 tablet_copy_service.cc:433] P 8df53dcdf65b407480e6d32372b48a35: ending tablet copy session 9cb2634fe3314e619359e41fbb351140-d7037feb51534b57b0ac70344cdc1e18 on tablet d7037feb51534b57b0ac70344cdc1e18 with peer 9cb2634fe3314e619359e41fbb351140
I20240503 08:12:54.206521 4752 raft_consensus.cc:1203] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20240503 08:12:54.766794 5073 raft_consensus.cc:1050] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35: attempting to promote NON_VOTER 9cb2634fe3314e619359e41fbb351140 to VOTER
I20240503 08:12:54.767585 5073 consensus_queue.cc:237] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 26, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:54.769665 4752 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 LEARNER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:12:54.769765 4355 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:12:54.770112 4430 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:12:54.770159 5072 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:54.770748 5072 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:54.771418 5036 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:54.785586 5075 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:54.786172 4430 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:54.789283 4355 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:54.789914 4752 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:54.791074 4182 catalog_manager.cc:5484] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: config changed from index 28 to 29, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:54.802855 4286 consensus_queue.cc:237] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:54.804472 4752 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:12:54.804868 4355 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:12:54.805518 5075 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:54.806735 5075 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:54.807329 5075 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:54.808110 4752 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:54.809602 4355 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:54.811992 4179 catalog_manager.cc:5484] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: config changed from index 29 to 30, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New cstate: current_term: 2 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:54.814782 4169 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d7037feb51534b57b0ac70344cdc1e18 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:54.815053 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet d7037feb51534b57b0ac70344cdc1e18 with delete_type TABLET_DATA_TOMBSTONED (TS 3523cd896bb842d68fbb6c15cdd1c3ed not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:34878
I20240503 08:12:54.815642 5083 tablet_replica.cc:330] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed: stopping tablet replica
I20240503 08:12:54.816358 5083 raft_consensus.cc:2229] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.816601 5083 raft_consensus.cc:2258] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.817301 5083 ts_tablet_manager.cc:1875] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:54.821547 5083 ts_tablet_manager.cc:1888] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240503 08:12:54.821635 5083 log.cc:1196] T d7037feb51534b57b0ac70344cdc1e18 P 3523cd896bb842d68fbb6c15cdd1c3ed: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-2-root/wals/d7037feb51534b57b0ac70344cdc1e18
I20240503 08:12:54.822158 4170 catalog_manager.cc:4830] TS 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567): tablet d7037feb51534b57b0ac70344cdc1e18 (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:54.991312 4238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:54.994529 4281 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
mode: GRACEFUL
new_leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
from {username='slave'} at 127.0.0.1:57228
I20240503 08:12:54.994709 4281 raft_consensus.cc:595] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 3 LEADER]: Received request to transfer leadership to TS 3523cd896bb842d68fbb6c15cdd1c3ed
I20240503 08:12:54.995610 4430 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819"
dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
mode: GRACEFUL
new_leader_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:34920
I20240503 08:12:54.995749 4430 raft_consensus.cc:595] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Received request to transfer leadership to TS 3482c85dd194402bbc0038963ed16d35
I20240503 08:12:54.996582 4355 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8ed061642594372a70df1d766d7b646"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
mode: GRACEFUL
new_leader_uuid: "8df53dcdf65b407480e6d32372b48a35"
from {username='slave'} at 127.0.0.1:50644
I20240503 08:12:54.996704 4355 raft_consensus.cc:595] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Received request to transfer leadership to TS 8df53dcdf65b407480e6d32372b48a35
I20240503 08:12:54.997051 4238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 08:12:54.997208 4238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:55.227555 5085 raft_consensus.cc:979] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed: : Instructing follower 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:55.227737 5085 raft_consensus.cc:1067] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Signalling peer 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:55.228376 4355 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:50592
I20240503 08:12:55.228533 4355 raft_consensus.cc:482] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:55.228619 4355 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:55.230414 4355 raft_consensus.cc:504] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:55.230952 4355 leader_election.cc:288] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 3 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:55.231248 4281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:55.231316 4430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:55.231426 4281 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:55.231545 4430 raft_consensus.cc:3032] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:12:55.231608 4430 raft_consensus.cc:729] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:55.231807 4430 consensus_queue.cc:260] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:55.232059 4430 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:55.233366 4281 raft_consensus.cc:2454] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 3.
I20240503 08:12:55.233722 4325 leader_election.cc:302] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:55.233901 4430 raft_consensus.cc:2454] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 3.
I20240503 08:12:55.233984 5068 raft_consensus.cc:2781] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:12:55.234362 5068 raft_consensus.cc:686] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Becoming Leader. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:55.234609 5068 consensus_queue.cc:237] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:55.236820 4179 catalog_manager.cc:5484] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: term changed from 2 to 3, leader changed from 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) to 3482c85dd194402bbc0038963ed16d35 (127.0.230.2). New cstate: current_term: 3 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:55.312494 5075 raft_consensus.cc:979] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35: : Instructing follower 3523cd896bb842d68fbb6c15cdd1c3ed to start an election
I20240503 08:12:55.312630 5075 raft_consensus.cc:1067] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 3 LEADER]: Signalling peer 3523cd896bb842d68fbb6c15cdd1c3ed to start an election
I20240503 08:12:55.313127 4430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240"
dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
from {username='slave'} at 127.0.0.1:34898
I20240503 08:12:55.313285 4430 raft_consensus.cc:482] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:55.313370 4430 raft_consensus.cc:3037] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:55.315415 4430 raft_consensus.cc:504] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:55.315938 4430 leader_election.cc:288] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 4 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:55.316262 4281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:55.316424 4281 raft_consensus.cc:3032] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:12:55.316362 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e50d2db8824a11af7a5e4f9f3ed240" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:55.316505 4281 raft_consensus.cc:729] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:55.316579 4355 raft_consensus.cc:3037] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:55.316717 4281 consensus_queue.cc:260] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:55.317024 4281 raft_consensus.cc:3037] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:55.319329 4355 raft_consensus.cc:2454] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 4.
I20240503 08:12:55.319497 4281 raft_consensus.cc:2454] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 4.
I20240503 08:12:55.319698 4400 leader_election.cc:302] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 3523cd896bb842d68fbb6c15cdd1c3ed; no voters:
I20240503 08:12:55.319895 5085 raft_consensus.cc:2781] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:12:55.320030 5085 raft_consensus.cc:686] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 LEADER]: Becoming Leader. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:55.320390 5085 consensus_queue.cc:237] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:55.322510 4179 catalog_manager.cc:5484] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: term changed from 3 to 4, leader changed from 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) to 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3). New cstate: current_term: 4 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:55.384452 5086 raft_consensus.cc:979] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35: : Instructing follower 8df53dcdf65b407480e6d32372b48a35 to start an election
I20240503 08:12:55.384610 5086 raft_consensus.cc:1067] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Signalling peer 8df53dcdf65b407480e6d32372b48a35 to start an election
I20240503 08:12:55.385214 4281 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8ed061642594372a70df1d766d7b646"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
from {username='slave'} at 127.0.0.1:57212
I20240503 08:12:55.385378 4281 raft_consensus.cc:482] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:55.385443 4281 raft_consensus.cc:3037] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:55.387569 4281 raft_consensus.cc:504] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:55.388056 4281 leader_election.cc:288] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 4 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:55.388447 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ed061642594372a70df1d766d7b646" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:55.388509 4430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ed061642594372a70df1d766d7b646" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:55.388620 4355 raft_consensus.cc:3032] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:12:55.388679 4355 raft_consensus.cc:729] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:55.388689 4430 raft_consensus.cc:3037] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:55.388934 4355 consensus_queue.cc:260] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:55.389189 4355 raft_consensus.cc:3037] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:55.390599 4430 raft_consensus.cc:2454] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 4.
I20240503 08:12:55.391039 4251 leader_election.cc:302] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:55.391155 4355 raft_consensus.cc:2454] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 4.
I20240503 08:12:55.391302 5075 raft_consensus.cc:2781] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:12:55.391433 5075 raft_consensus.cc:686] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 4 LEADER]: Becoming Leader. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:55.391709 5075 consensus_queue.cc:237] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:55.393850 4179 catalog_manager.cc:5484] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: term changed from 3 to 4, leader changed from 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) to 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1). New cstate: current_term: 4 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:55.658471 4281 raft_consensus.cc:1261] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:12:55.659010 5086 consensus_queue.cc:1035] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:55.662786 4430 raft_consensus.cc:1261] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:12:55.663239 5068 consensus_queue.cc:1035] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:55.766108 4355 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240503 08:12:55.766634 5085 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:55.770068 4281 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240503 08:12:55.770532 5097 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:55.811993 4355 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240503 08:12:55.812518 5075 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:55.815717 4430 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240503 08:12:55.816236 5073 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:55.832257 4678 consensus_queue.cc:237] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:12:55.834549 4281 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Refusing update from remote peer 23c12ef68ccc4fad80eb918f77be34e2: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 08:12:55.835026 4355 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Refusing update from remote peer 23c12ef68ccc4fad80eb918f77be34e2: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 08:12:55.835038 5105 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:12:55.835567 5022 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:12:55.838151 5022 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 30 to 32, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:12:55.838555 4286 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 30 to 32, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:12:55.839618 4355 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 30 to 32, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
W20240503 08:12:55.844125 4642 consensus_peers.cc:485] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 -> Peer 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6:39275): Couldn't send request to peer 981e135f98fb47a1afc7fda11da0e834. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2b69fcfaa7742919bc05385b49623dd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:55.844019 4181 catalog_manager.cc:5484] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: config changed from index 30 to 32, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New cstate: current_term: 3 leader_uuid: "23c12ef68ccc4fad80eb918f77be34e2" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:12:55.844614 4430 consensus_queue.cc:237] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:12:55.846542 4286 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 08:12:55.847043 4355 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 08:12:55.847543 5097 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:55.847872 5070 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:55.849932 5097 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 LEADER]: Committing config change with OpId 4.30: config changed from index -1 to 30, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:12:55.850528 4286 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index -1 to 30, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:12:55.853016 4355 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index -1 to 30, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
W20240503 08:12:55.855522 4401 consensus_peers.cc:485] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed -> Peer 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4:32895): Couldn't send request to peer 23c12ef68ccc4fad80eb918f77be34e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03e50d2db8824a11af7a5e4f9f3ed240. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:55.856213 4281 consensus_queue.cc:237] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:12:55.858402 4361 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 08:12:55.858420 4179 catalog_manager.cc:5484] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: config changed from index -1 to 30, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New cstate: current_term: 4 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:55.864070 5075 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:55.864683 4430 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 08:12:55.865195 5075 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:55.867162 5073 raft_consensus.cc:2932] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 4 LEADER]: Committing config change with OpId 4.30: config changed from index -1 to 30, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:12:55.871594 4430 raft_consensus.cc:2932] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index -1 to 30, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:12:55.872673 4184 catalog_manager.cc:5484] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: config changed from index -1 to 30, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New cstate: current_term: 4 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:55.874545 4361 raft_consensus.cc:2932] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index -1 to 30, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:12:55.874996 4355 consensus_queue.cc:237] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
W20240503 08:12:55.878281 4251 consensus_peers.cc:485] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 -> Peer 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6:39275): Couldn't send request to peer 981e135f98fb47a1afc7fda11da0e834. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8ed061642594372a70df1d766d7b646. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:55.878674 4430 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 08:12:55.879343 5086 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:55.880234 4286 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 08:12:55.881050 5086 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:55.883354 5068 raft_consensus.cc:2932] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
W20240503 08:12:55.884003 4325 consensus_peers.cc:485] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 -> Peer 9cb2634fe3314e619359e41fbb351140 (127.0.230.5:40387): Couldn't send request to peer 9cb2634fe3314e619359e41fbb351140. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1440f42da8514e618147c69fa2795415. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:55.883980 4430 raft_consensus.cc:2932] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:55.884166 4286 raft_consensus.cc:2932] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:55.888051 4184 catalog_manager.cc:5484] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: config changed from index -1 to 29, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New cstate: current_term: 3 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:55.889528 4281 consensus_queue.cc:237] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:12:55.892016 4752 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
W20240503 08:12:55.892235 4252 consensus_peers.cc:485] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 -> Peer 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4:32895): Couldn't send request to peer 23c12ef68ccc4fad80eb918f77be34e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7037feb51534b57b0ac70344cdc1e18. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:55.892457 4361 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:12:55.892596 5100 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:55.893545 5100 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:55.895174 5075 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:12:55.896402 4752 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:12:55.896473 4361 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:12:55.900194 4355 consensus_queue.cc:237] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:55.900121 4181 catalog_manager.cc:5484] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: config changed from index 30 to 31, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New cstate: current_term: 2 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:55.902634 4430 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 08:12:55.902630 4826 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 08:12:55.903155 5068 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:55.903429 5092 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240503 08:12:55.904050 4325 consensus_peers.cc:485] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 -> Peer 9cb2634fe3314e619359e41fbb351140 (127.0.230.5:40387): Couldn't send request to peer 9cb2634fe3314e619359e41fbb351140. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c94415b8c11d4f9790344c93eff4c331. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:55.908427 5068 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:55.908980 4826 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:55.909505 4430 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:12:55.913010 4179 catalog_manager.cc:5484] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: config changed from index 29 to 30, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New cstate: current_term: 1 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:56.268254 5124 ts_tablet_manager.cc:902] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2: Initiating tablet copy from peer 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:56.270483 5124 tablet_copy_client.cc:308] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.3:38567
I20240503 08:12:56.275997 4446 tablet_copy_service.cc:140] P 3523cd896bb842d68fbb6c15cdd1c3ed: Received BeginTabletCopySession request for tablet 03e50d2db8824a11af7a5e4f9f3ed240 from peer 23c12ef68ccc4fad80eb918f77be34e2 ({username='slave'} at 127.0.0.1:57524)
I20240503 08:12:56.276103 4446 tablet_copy_service.cc:161] P 3523cd896bb842d68fbb6c15cdd1c3ed: Beginning new tablet copy session on tablet 03e50d2db8824a11af7a5e4f9f3ed240 from peer 23c12ef68ccc4fad80eb918f77be34e2 at {username='slave'} at 127.0.0.1:57524: session id = 23c12ef68ccc4fad80eb918f77be34e2-03e50d2db8824a11af7a5e4f9f3ed240
I20240503 08:12:56.278281 4446 tablet_copy_source_session.cc:205] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:56.279230 5124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e50d2db8824a11af7a5e4f9f3ed240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.284807 5124 tablet_copy_client.cc:783] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:56.285001 5124 tablet_copy_client.cc:647] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:56.286377 5128 tablet_copy_client.cc:1002] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:56.296191 5130 ts_tablet_manager.cc:902] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140: Initiating tablet copy from peer 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:56.296612 5131 ts_tablet_manager.cc:902] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834: Initiating tablet copy from peer 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947)
I20240503 08:12:56.298475 5131 tablet_copy_client.cc:308] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.1:32947
I20240503 08:12:56.303884 5130 tablet_copy_client.cc:308] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.2:44611
I20240503 08:12:56.313566 5144 ts_tablet_manager.cc:902] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2: Initiating tablet copy from peer 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947)
I20240503 08:12:56.315714 5144 tablet_copy_client.cc:308] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.1:32947
I20240503 08:12:56.316680 4297 tablet_copy_service.cc:140] P 8df53dcdf65b407480e6d32372b48a35: Received BeginTabletCopySession request for tablet d7037feb51534b57b0ac70344cdc1e18 from peer 23c12ef68ccc4fad80eb918f77be34e2 ({username='slave'} at 127.0.0.1:57252)
I20240503 08:12:56.316805 4297 tablet_copy_service.cc:161] P 8df53dcdf65b407480e6d32372b48a35: Beginning new tablet copy session on tablet d7037feb51534b57b0ac70344cdc1e18 from peer 23c12ef68ccc4fad80eb918f77be34e2 at {username='slave'} at 127.0.0.1:57252: session id = 23c12ef68ccc4fad80eb918f77be34e2-d7037feb51534b57b0ac70344cdc1e18
I20240503 08:12:56.319897 4297 tablet_copy_source_session.cc:205] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:56.320281 4372 tablet_copy_service.cc:140] P 3482c85dd194402bbc0038963ed16d35: Received BeginTabletCopySession request for tablet 1440f42da8514e618147c69fa2795415 from peer 9cb2634fe3314e619359e41fbb351140 ({username='slave'} at 127.0.0.1:38274)
I20240503 08:12:56.320387 4372 tablet_copy_service.cc:161] P 3482c85dd194402bbc0038963ed16d35: Beginning new tablet copy session on tablet 1440f42da8514e618147c69fa2795415 from peer 9cb2634fe3314e619359e41fbb351140 at {username='slave'} at 127.0.0.1:38274: session id = 9cb2634fe3314e619359e41fbb351140-1440f42da8514e618147c69fa2795415
I20240503 08:12:56.321153 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7037feb51534b57b0ac70344cdc1e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.322405 4372 tablet_copy_source_session.cc:205] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:56.323277 4297 tablet_copy_service.cc:140] P 8df53dcdf65b407480e6d32372b48a35: Received BeginTabletCopySession request for tablet f8ed061642594372a70df1d766d7b646 from peer 981e135f98fb47a1afc7fda11da0e834 ({username='slave'} at 127.0.0.1:40334)
I20240503 08:12:56.323381 5130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1440f42da8514e618147c69fa2795415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.323467 4297 tablet_copy_service.cc:161] P 8df53dcdf65b407480e6d32372b48a35: Beginning new tablet copy session on tablet f8ed061642594372a70df1d766d7b646 from peer 981e135f98fb47a1afc7fda11da0e834 at {username='slave'} at 127.0.0.1:40334: session id = 981e135f98fb47a1afc7fda11da0e834-f8ed061642594372a70df1d766d7b646
I20240503 08:12:56.326807 4297 tablet_copy_source_session.cc:205] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:56.327796 5131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ed061642594372a70df1d766d7b646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.328789 5130 tablet_copy_client.cc:783] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:56.329035 5130 tablet_copy_client.cc:647] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:56.330004 5144 tablet_copy_client.cc:783] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:56.330320 5144 tablet_copy_client.cc:647] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:56.331928 5148 tablet_copy_client.cc:1002] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:56.333477 5131 tablet_copy_client.cc:783] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:56.333655 5131 tablet_copy_client.cc:647] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:56.334508 5141 tablet_copy_client.cc:1002] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:56.335108 5136 tablet_copy_client.cc:1002] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:56.344362 5149 ts_tablet_manager.cc:902] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140: Initiating tablet copy from peer 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:56.351866 5149 tablet_copy_client.cc:308] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.2:44611
I20240503 08:12:56.352412 4372 tablet_copy_service.cc:140] P 3482c85dd194402bbc0038963ed16d35: Received BeginTabletCopySession request for tablet c94415b8c11d4f9790344c93eff4c331 from peer 9cb2634fe3314e619359e41fbb351140 ({username='slave'} at 127.0.0.1:38274)
I20240503 08:12:56.352530 4372 tablet_copy_service.cc:161] P 3482c85dd194402bbc0038963ed16d35: Beginning new tablet copy session on tablet c94415b8c11d4f9790344c93eff4c331 from peer 9cb2634fe3314e619359e41fbb351140 at {username='slave'} at 127.0.0.1:38274: session id = 9cb2634fe3314e619359e41fbb351140-c94415b8c11d4f9790344c93eff4c331
I20240503 08:12:56.354740 4372 tablet_copy_source_session.cc:205] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:56.355777 5149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94415b8c11d4f9790344c93eff4c331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.361927 5149 tablet_copy_client.cc:783] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:56.362116 5149 tablet_copy_client.cc:647] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:56.363210 5153 tablet_copy_client.cc:1002] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:56.375139 5155 ts_tablet_manager.cc:902] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834: Initiating tablet copy from peer 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4:32895)
I20240503 08:12:56.382656 5155 tablet_copy_client.cc:308] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.4:32895
I20240503 08:12:56.388324 4688 tablet_copy_service.cc:140] P 23c12ef68ccc4fad80eb918f77be34e2: Received BeginTabletCopySession request for tablet b2b69fcfaa7742919bc05385b49623dd from peer 981e135f98fb47a1afc7fda11da0e834 ({username='slave'} at 127.0.0.1:59354)
I20240503 08:12:56.388505 4688 tablet_copy_service.cc:161] P 23c12ef68ccc4fad80eb918f77be34e2: Beginning new tablet copy session on tablet b2b69fcfaa7742919bc05385b49623dd from peer 981e135f98fb47a1afc7fda11da0e834 at {username='slave'} at 127.0.0.1:59354: session id = 981e135f98fb47a1afc7fda11da0e834-b2b69fcfaa7742919bc05385b49623dd
I20240503 08:12:56.390909 4688 tablet_copy_source_session.cc:205] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:56.391930 5155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b69fcfaa7742919bc05385b49623dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.396605 5155 tablet_copy_client.cc:783] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:56.396829 5155 tablet_copy_client.cc:647] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:56.398098 5159 tablet_copy_client.cc:1002] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Injecting latency into file download: 2000
I20240503 08:12:56.997491 4238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:57.000330 4286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59aedfe83feb48d799de8eda2afb1860"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
mode: GRACEFUL
new_leader_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:57228
I20240503 08:12:57.000542 4286 raft_consensus.cc:595] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Received request to transfer leadership to TS 3482c85dd194402bbc0038963ed16d35
I20240503 08:12:57.001364 4355 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
mode: GRACEFUL
new_leader_uuid: "8df53dcdf65b407480e6d32372b48a35"
from {username='slave'} at 127.0.0.1:50644
I20240503 08:12:57.001521 4355 raft_consensus.cc:595] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Received request to transfer leadership to TS 8df53dcdf65b407480e6d32372b48a35
I20240503 08:12:57.002403 4432 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4"
dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
mode: GRACEFUL
new_leader_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:34920
I20240503 08:12:57.002533 4432 raft_consensus.cc:595] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Received request to transfer leadership to TS 3482c85dd194402bbc0038963ed16d35
I20240503 08:12:57.002928 4238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 08:12:57.003140 4238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:57.069558 5073 raft_consensus.cc:979] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35: : Instructing follower 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:57.069753 5162 raft_consensus.cc:1067] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Signalling peer 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:57.070308 4355 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "59aedfe83feb48d799de8eda2afb1860"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:50608
I20240503 08:12:57.070590 4355 raft_consensus.cc:482] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:57.070758 4355 raft_consensus.cc:3037] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:57.072845 4355 raft_consensus.cc:504] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:57.073386 4355 leader_election.cc:288] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 2 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4:32895)
I20240503 08:12:57.073720 4677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59aedfe83feb48d799de8eda2afb1860" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "23c12ef68ccc4fad80eb918f77be34e2"
I20240503 08:12:57.073886 4677 raft_consensus.cc:3037] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:57.076093 4677 raft_consensus.cc:2454] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 2.
I20240503 08:12:57.076349 4286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59aedfe83feb48d799de8eda2afb1860" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:57.076503 4286 raft_consensus.cc:3032] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:57.076469 4327 leader_election.cc:302] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c12ef68ccc4fad80eb918f77be34e2, 3482c85dd194402bbc0038963ed16d35; no voters:
I20240503 08:12:57.076579 4286 raft_consensus.cc:729] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:57.076740 5086 raft_consensus.cc:2781] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:57.076887 5086 raft_consensus.cc:686] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Becoming Leader. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:57.076887 4286 consensus_queue.cc:260] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:57.077237 4286 raft_consensus.cc:3037] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:57.077167 5086 consensus_queue.cc:237] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:57.079346 4286 raft_consensus.cc:2454] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 2.
I20240503 08:12:57.081154 4181 catalog_manager.cc:5484] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: term changed from 1 to 2, leader changed from 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) to 3482c85dd194402bbc0038963ed16d35 (127.0.230.2). New cstate: current_term: 2 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:57.131722 5165 raft_consensus.cc:979] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed: : Instructing follower 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:57.131868 5165 raft_consensus.cc:1067] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Signalling peer 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:57.132445 4355 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:50592
I20240503 08:12:57.132611 4355 raft_consensus.cc:482] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:57.132709 4355 raft_consensus.cc:3037] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:57.135089 4355 raft_consensus.cc:504] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:57.135790 4355 leader_election.cc:288] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 3 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:57.136180 4281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:57.136251 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:57.136343 4281 raft_consensus.cc:3037] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:57.136448 4432 raft_consensus.cc:3032] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:12:57.136526 4432 raft_consensus.cc:729] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:57.136725 4432 consensus_queue.cc:260] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:57.136996 4432 raft_consensus.cc:3037] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:57.138252 4281 raft_consensus.cc:2454] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 3.
I20240503 08:12:57.138563 4325 leader_election.cc:302] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:57.138790 5086 raft_consensus.cc:2781] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:12:57.138790 4432 raft_consensus.cc:2454] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 3.
I20240503 08:12:57.138967 5086 raft_consensus.cc:686] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Becoming Leader. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:57.139176 5086 consensus_queue.cc:237] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:57.141541 4181 catalog_manager.cc:5484] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: term changed from 2 to 3, leader changed from 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) to 3482c85dd194402bbc0038963ed16d35 (127.0.230.2). New cstate: current_term: 3 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:57.148995 5068 raft_consensus.cc:979] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35: : Instructing follower 8df53dcdf65b407480e6d32372b48a35 to start an election
I20240503 08:12:57.149130 5068 raft_consensus.cc:1067] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Signalling peer 8df53dcdf65b407480e6d32372b48a35 to start an election
I20240503 08:12:57.149622 4281 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
from {username='slave'} at 127.0.0.1:57212
I20240503 08:12:57.149786 4281 raft_consensus.cc:482] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:57.149870 4281 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:57.151978 4281 raft_consensus.cc:504] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:57.152524 4281 leader_election.cc:288] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 4 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:57.152868 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:57.153052 4355 raft_consensus.cc:3032] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:12:57.153023 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:57.153124 4355 raft_consensus.cc:729] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:57.153223 4432 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:57.153357 4355 consensus_queue.cc:260] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:57.153671 4355 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:57.155791 4432 raft_consensus.cc:2454] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 4.
I20240503 08:12:57.155978 4355 raft_consensus.cc:2454] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 4.
I20240503 08:12:57.156069 4251 leader_election.cc:302] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3523cd896bb842d68fbb6c15cdd1c3ed, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:57.156315 5162 raft_consensus.cc:2781] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:12:57.156424 5162 raft_consensus.cc:686] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 4 LEADER]: Becoming Leader. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:57.156630 5162 consensus_queue.cc:237] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:57.159138 4181 catalog_manager.cc:5484] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: term changed from 3 to 4, leader changed from 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) to 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1). New cstate: current_term: 4 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:57.531303 4677 raft_consensus.cc:1261] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:12:57.531792 5068 consensus_queue.cc:1035] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:57.536087 4281 raft_consensus.cc:1261] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:12:57.536684 5068 consensus_queue.cc:1035] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:57.616811 4355 raft_consensus.cc:1261] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240503 08:12:57.617262 5161 consensus_queue.cc:1035] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:57.620713 4432 raft_consensus.cc:1261] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240503 08:12:57.621134 5162 consensus_queue.cc:1035] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:57.674533 4281 raft_consensus.cc:1261] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:12:57.675040 5068 consensus_queue.cc:1035] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:57.678825 4432 raft_consensus.cc:1261] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:12:57.679454 5092 consensus_queue.cc:1035] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:12:58.287144 5124 tablet_copy_client.cc:523] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:58.290319 5124 tablet_bootstrap.cc:492] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2: Bootstrap starting.
I20240503 08:12:58.331851 5124 tablet_bootstrap.cc:492] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=112 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:58.332187 5124 tablet_bootstrap.cc:492] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2: Bootstrap complete.
I20240503 08:12:58.332439 5124 ts_tablet_manager.cc:1367] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2: Time spent bootstrapping tablet: real 0.042s user 0.034s sys 0.009s
I20240503 08:12:58.333155 5144 tablet_copy_client.cc:523] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:58.333186 5124 raft_consensus.cc:348] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:12:58.333386 5124 raft_consensus.cc:729] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 23c12ef68ccc4fad80eb918f77be34e2, State: Initialized, Role: LEARNER
I20240503 08:12:58.333554 5124 consensus_queue.cc:260] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:12:58.334241 5124 ts_tablet_manager.cc:1398] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:58.334899 4446 tablet_copy_service.cc:342] P 3523cd896bb842d68fbb6c15cdd1c3ed: Request end of tablet copy session 23c12ef68ccc4fad80eb918f77be34e2-03e50d2db8824a11af7a5e4f9f3ed240 received from {username='slave'} at 127.0.0.1:57524
I20240503 08:12:58.334995 4446 tablet_copy_service.cc:433] P 3523cd896bb842d68fbb6c15cdd1c3ed: ending tablet copy session 23c12ef68ccc4fad80eb918f77be34e2-03e50d2db8824a11af7a5e4f9f3ed240 on tablet 03e50d2db8824a11af7a5e4f9f3ed240 with peer 23c12ef68ccc4fad80eb918f77be34e2
I20240503 08:12:58.335964 5130 tablet_copy_client.cc:523] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:58.336531 5144 tablet_bootstrap.cc:492] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2: Bootstrap starting.
I20240503 08:12:58.337814 5131 tablet_copy_client.cc:523] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:58.339090 5130 tablet_bootstrap.cc:492] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140: Bootstrap starting.
I20240503 08:12:58.340479 5131 tablet_bootstrap.cc:492] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834: Bootstrap starting.
I20240503 08:12:58.363674 5149 tablet_copy_client.cc:523] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:58.367102 5149 tablet_bootstrap.cc:492] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140: Bootstrap starting.
I20240503 08:12:58.378279 5144 tablet_bootstrap.cc:492] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=95 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:58.378762 5144 tablet_bootstrap.cc:492] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2: Bootstrap complete.
I20240503 08:12:58.379243 5144 ts_tablet_manager.cc:1367] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2: Time spent bootstrapping tablet: real 0.043s user 0.040s sys 0.001s
I20240503 08:12:58.380062 5144 raft_consensus.cc:348] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:12:58.380419 5144 raft_consensus.cc:729] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 23c12ef68ccc4fad80eb918f77be34e2, State: Initialized, Role: LEARNER
I20240503 08:12:58.380702 5144 consensus_queue.cc:260] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:12:58.383558 5144 ts_tablet_manager.cc:1398] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:58.386219 4297 tablet_copy_service.cc:342] P 8df53dcdf65b407480e6d32372b48a35: Request end of tablet copy session 23c12ef68ccc4fad80eb918f77be34e2-d7037feb51534b57b0ac70344cdc1e18 received from {username='slave'} at 127.0.0.1:57252
I20240503 08:12:58.386314 4297 tablet_copy_service.cc:433] P 8df53dcdf65b407480e6d32372b48a35: ending tablet copy session 23c12ef68ccc4fad80eb918f77be34e2-d7037feb51534b57b0ac70344cdc1e18 on tablet d7037feb51534b57b0ac70344cdc1e18 with peer 23c12ef68ccc4fad80eb918f77be34e2
I20240503 08:12:58.392756 5130 tablet_bootstrap.cc:492] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=96 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:58.393134 5130 tablet_bootstrap.cc:492] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140: Bootstrap complete.
I20240503 08:12:58.393414 5130 ts_tablet_manager.cc:1367] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140: Time spent bootstrapping tablet: real 0.054s user 0.042s sys 0.004s
I20240503 08:12:58.394520 5130 raft_consensus.cc:348] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:58.394805 5130 raft_consensus.cc:729] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 9cb2634fe3314e619359e41fbb351140, State: Initialized, Role: LEARNER
I20240503 08:12:58.395017 5130 consensus_queue.cc:260] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:58.395939 5130 ts_tablet_manager.cc:1398] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:58.397403 4372 tablet_copy_service.cc:342] P 3482c85dd194402bbc0038963ed16d35: Request end of tablet copy session 9cb2634fe3314e619359e41fbb351140-1440f42da8514e618147c69fa2795415 received from {username='slave'} at 127.0.0.1:38274
I20240503 08:12:58.397511 4372 tablet_copy_service.cc:433] P 3482c85dd194402bbc0038963ed16d35: ending tablet copy session 9cb2634fe3314e619359e41fbb351140-1440f42da8514e618147c69fa2795415 on tablet 1440f42da8514e618147c69fa2795415 with peer 9cb2634fe3314e619359e41fbb351140
I20240503 08:12:58.398599 5155 tablet_copy_client.cc:523] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:58.399753 5131 tablet_bootstrap.cc:492] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=85 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:58.400136 5131 tablet_bootstrap.cc:492] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834: Bootstrap complete.
I20240503 08:12:58.400350 5131 ts_tablet_manager.cc:1367] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834: Time spent bootstrapping tablet: real 0.060s user 0.033s sys 0.009s
I20240503 08:12:58.400964 5131 raft_consensus.cc:348] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:12:58.401165 5131 raft_consensus.cc:729] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 981e135f98fb47a1afc7fda11da0e834, State: Initialized, Role: LEARNER
I20240503 08:12:58.401450 5155 tablet_bootstrap.cc:492] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834: Bootstrap starting.
I20240503 08:12:58.401348 5131 consensus_queue.cc:260] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:12:58.401821 5131 ts_tablet_manager.cc:1398] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:58.402443 4297 tablet_copy_service.cc:342] P 8df53dcdf65b407480e6d32372b48a35: Request end of tablet copy session 981e135f98fb47a1afc7fda11da0e834-f8ed061642594372a70df1d766d7b646 received from {username='slave'} at 127.0.0.1:40334
I20240503 08:12:58.402550 4297 tablet_copy_service.cc:433] P 8df53dcdf65b407480e6d32372b48a35: ending tablet copy session 981e135f98fb47a1afc7fda11da0e834-f8ed061642594372a70df1d766d7b646 on tablet f8ed061642594372a70df1d766d7b646 with peer 981e135f98fb47a1afc7fda11da0e834
I20240503 08:12:58.418140 4678 raft_consensus.cc:1203] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.29->[4.30-4.30] Dedup: 4.30->[]
I20240503 08:12:58.429736 5149 tablet_bootstrap.cc:492] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=107 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:58.430132 5149 tablet_bootstrap.cc:492] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140: Bootstrap complete.
I20240503 08:12:58.430361 5149 ts_tablet_manager.cc:1367] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140: Time spent bootstrapping tablet: real 0.063s user 0.030s sys 0.008s
I20240503 08:12:58.431059 5149 raft_consensus.cc:348] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:58.431299 5149 raft_consensus.cc:729] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9cb2634fe3314e619359e41fbb351140, State: Initialized, Role: LEARNER
I20240503 08:12:58.431504 5149 consensus_queue.cc:260] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
I20240503 08:12:58.432102 5149 ts_tablet_manager.cc:1398] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:12:58.432668 4372 tablet_copy_service.cc:342] P 3482c85dd194402bbc0038963ed16d35: Request end of tablet copy session 9cb2634fe3314e619359e41fbb351140-c94415b8c11d4f9790344c93eff4c331 received from {username='slave'} at 127.0.0.1:38274
I20240503 08:12:58.432770 4372 tablet_copy_service.cc:433] P 3482c85dd194402bbc0038963ed16d35: ending tablet copy session 9cb2634fe3314e619359e41fbb351140-c94415b8c11d4f9790344c93eff4c331 on tablet c94415b8c11d4f9790344c93eff4c331 with peer 9cb2634fe3314e619359e41fbb351140
I20240503 08:12:58.444242 5155 tablet_bootstrap.cc:492] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=107 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:58.444609 5155 tablet_bootstrap.cc:492] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834: Bootstrap complete.
I20240503 08:12:58.444844 5155 ts_tablet_manager.cc:1367] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834: Time spent bootstrapping tablet: real 0.043s user 0.038s sys 0.003s
I20240503 08:12:58.445530 5155 raft_consensus.cc:348] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:12:58.445775 5155 raft_consensus.cc:729] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 981e135f98fb47a1afc7fda11da0e834, State: Initialized, Role: LEARNER
I20240503 08:12:58.445983 5155 consensus_queue.cc:260] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:12:58.446493 5155 ts_tablet_manager.cc:1398] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:58.447088 4688 tablet_copy_service.cc:342] P 23c12ef68ccc4fad80eb918f77be34e2: Request end of tablet copy session 981e135f98fb47a1afc7fda11da0e834-b2b69fcfaa7742919bc05385b49623dd received from {username='slave'} at 127.0.0.1:59354
I20240503 08:12:58.447185 4688 tablet_copy_service.cc:433] P 23c12ef68ccc4fad80eb918f77be34e2: ending tablet copy session 981e135f98fb47a1afc7fda11da0e834-b2b69fcfaa7742919bc05385b49623dd on tablet b2b69fcfaa7742919bc05385b49623dd with peer 981e135f98fb47a1afc7fda11da0e834
I20240503 08:12:58.597615 4826 raft_consensus.cc:1203] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.29->[4.30-4.30] Dedup: 4.30->[]
I20240503 08:12:58.663830 4752 raft_consensus.cc:1203] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.28->[3.29-3.29] Dedup: 3.29->[]
I20240503 08:12:58.676157 4826 raft_consensus.cc:1203] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.31->[3.32-3.32] Dedup: 3.32->[]
I20240503 08:12:58.819782 5193 raft_consensus.cc:1050] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed: attempting to promote NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 to VOTER
I20240503 08:12:58.820485 5193 consensus_queue.cc:237] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 28, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:58.822372 4678 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2 [term 4 LEARNER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:12:58.822425 4355 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:12:58.822803 5193 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:58.822893 4281 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:12:58.823261 5192 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:58.823557 5193 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:58.826140 5174 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index 30 to 31, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:58.827227 4678 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:58.828910 4355 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:58.832517 4184 catalog_manager.cc:5484] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2 reported cstate change: config changed from index 30 to 31, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:58.833112 4281 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:58.842543 4432 consensus_queue.cc:237] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:58.844044 4355 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 08:12:58.844120 4678 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 08:12:58.844722 5192 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:12:58.845389 5193 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:12:58.847379 5174 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:58.847963 4678 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:58.850864 4170 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03e50d2db8824a11af7a5e4f9f3ed240 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:58.851877 4181 catalog_manager.cc:5484] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: config changed from index 31 to 32, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New cstate: current_term: 4 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:58.852449 4355 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:58.859220 4273 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e50d2db8824a11af7a5e4f9f3ed240 with delete_type TABLET_DATA_TOMBSTONED (TS 8df53dcdf65b407480e6d32372b48a35 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:57198
I20240503 08:12:58.859788 5202 tablet_replica.cc:330] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35: stopping tablet replica
I20240503 08:12:58.860105 5202 raft_consensus.cc:2229] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:58.860826 5202 raft_consensus.cc:2258] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:58.861510 5202 ts_tablet_manager.cc:1875] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:58.862361 4752 raft_consensus.cc:1203] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.29->[1.30-1.30] Dedup: 1.30->[]
I20240503 08:12:58.867084 5202 ts_tablet_manager.cc:1888] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.31
I20240503 08:12:58.867173 5202 log.cc:1196] T 03e50d2db8824a11af7a5e4f9f3ed240 P 8df53dcdf65b407480e6d32372b48a35: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-0-root/wals/03e50d2db8824a11af7a5e4f9f3ed240
I20240503 08:12:58.867787 4169 catalog_manager.cc:4830] TS 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947): tablet 03e50d2db8824a11af7a5e4f9f3ed240 (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:58.868638 4678 raft_consensus.cc:1203] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.30->[2.31-2.31] Dedup: 2.31->[]
I20240503 08:12:59.003517 4238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:12:59.006714 4281 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
mode: GRACEFUL
new_leader_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:57228
I20240503 08:12:59.006985 4281 raft_consensus.cc:595] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 4 LEADER]: Received request to transfer leadership to TS 3482c85dd194402bbc0038963ed16d35
I20240503 08:12:59.008003 4355 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
mode: GRACEFUL
new_leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
from {username='slave'} at 127.0.0.1:50644
I20240503 08:12:59.008191 4355 raft_consensus.cc:595] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Received request to transfer leadership to TS 3523cd896bb842d68fbb6c15cdd1c3ed
I20240503 08:12:59.008562 4238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 08:12:59.008780 4238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:12:59.066123 5086 raft_consensus.cc:979] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35: : Instructing follower 3523cd896bb842d68fbb6c15cdd1c3ed to start an election
I20240503 08:12:59.066282 5086 raft_consensus.cc:1067] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Signalling peer 3523cd896bb842d68fbb6c15cdd1c3ed to start an election
I20240503 08:12:59.066773 4432 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4"
dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
from {username='slave'} at 127.0.0.1:34888
I20240503 08:12:59.066939 4432 raft_consensus.cc:482] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:59.067028 4432 raft_consensus.cc:3037] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:59.070045 4432 raft_consensus.cc:504] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:59.070731 4432 leader_election.cc:288] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 4 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:12:59.071180 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:12:59.071353 4355 raft_consensus.cc:3032] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:12:59.071432 4355 raft_consensus.cc:729] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:59.071678 4355 consensus_queue.cc:260] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:59.071980 4355 raft_consensus.cc:3037] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:59.072299 4281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82e1e5d27e441ccbd62d1296c06a2f4" candidate_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:59.072432 4281 raft_consensus.cc:3037] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:59.074206 5204 raft_consensus.cc:1050] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2: attempting to promote NON_VOTER 981e135f98fb47a1afc7fda11da0e834 to VOTER
I20240503 08:12:59.075006 5204 consensus_queue.cc:237] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 30, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } }
I20240503 08:12:59.076141 4355 raft_consensus.cc:2454] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 4.
I20240503 08:12:59.076500 4400 leader_election.cc:302] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 3523cd896bb842d68fbb6c15cdd1c3ed; no voters:
I20240503 08:12:59.076788 5193 raft_consensus.cc:2781] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:12:59.076890 5193 raft_consensus.cc:686] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 LEADER]: Becoming Leader. State: Replica: 3523cd896bb842d68fbb6c15cdd1c3ed, State: Running, Role: LEADER
I20240503 08:12:59.077090 5193 consensus_queue.cc:237] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:59.077260 4281 raft_consensus.cc:2454] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3523cd896bb842d68fbb6c15cdd1c3ed in term 4.
I20240503 08:12:59.077595 4826 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834 [term 3 LEARNER]: Refusing update from remote peer 23c12ef68ccc4fad80eb918f77be34e2: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 08:12:59.077745 4355 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Refusing update from remote peer 23c12ef68ccc4fad80eb918f77be34e2: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 08:12:59.078078 5163 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:12:59.078621 5163 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:12:59.079470 4281 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Refusing update from remote peer 23c12ef68ccc4fad80eb918f77be34e2: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 08:12:59.079869 5204 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:12:59.080762 4179 catalog_manager.cc:5484] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: term changed from 3 to 4, leader changed from 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) to 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3). New cstate: current_term: 4 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:59.082427 5205 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.083007 4355 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.087616 4826 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.088523 4281 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.087213 4179 catalog_manager.cc:5484] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 reported cstate change: config changed from index 32 to 33, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "23c12ef68ccc4fad80eb918f77be34e2" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:59.095175 4678 consensus_queue.cc:237] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } }
I20240503 08:12:59.096957 4826 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834 [term 3 FOLLOWER]: Refusing update from remote peer 23c12ef68ccc4fad80eb918f77be34e2: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240503 08:12:59.097059 4281 raft_consensus.cc:1261] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Refusing update from remote peer 23c12ef68ccc4fad80eb918f77be34e2: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240503 08:12:59.097493 5163 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:12:59.097987 5163 consensus_queue.cc:1035] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:12:59.099320 5211 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.099948 4286 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.104091 4231 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b2b69fcfaa7742919bc05385b49623dd with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:59.104339 5203 raft_consensus.cc:1050] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35: attempting to promote NON_VOTER 9cb2634fe3314e619359e41fbb351140 to VOTER
I20240503 08:12:59.104538 4181 catalog_manager.cc:5484] T b2b69fcfaa7742919bc05385b49623dd P 23c12ef68ccc4fad80eb918f77be34e2 reported cstate change: config changed from index 33 to 34, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New cstate: current_term: 3 leader_uuid: "23c12ef68ccc4fad80eb918f77be34e2" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:59.105046 5203 consensus_queue.cc:237] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:59.107955 4432 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 08:12:59.108013 4826 raft_consensus.cc:2932] T b2b69fcfaa7742919bc05385b49623dd P 981e135f98fb47a1afc7fda11da0e834 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.108482 4752 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140 [term 3 LEARNER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 08:12:59.108445 5086 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:59.108907 5086 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:59.109261 4351 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b69fcfaa7742919bc05385b49623dd with delete_type TABLET_DATA_TOMBSTONED (TS 3482c85dd194402bbc0038963ed16d35 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:50578
I20240503 08:12:59.122022 5086 raft_consensus.cc:2932] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.122601 4432 raft_consensus.cc:2932] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.123296 5215 tablet_replica.cc:330] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35: stopping tablet replica
I20240503 08:12:59.123489 5215 raft_consensus.cc:2229] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:59.123731 5215 raft_consensus.cc:2258] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:59.124689 5215 ts_tablet_manager.cc:1875] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:59.127077 4184 catalog_manager.cc:5484] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: config changed from index 29 to 30, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.127669 5218 raft_consensus.cc:1050] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35: attempting to promote NON_VOTER 981e135f98fb47a1afc7fda11da0e834 to VOTER
I20240503 08:12:59.128381 5218 consensus_queue.cc:237] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 28, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } }
I20240503 08:12:59.131305 4286 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 08:12:59.132232 5220 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:59.132289 4752 raft_consensus.cc:2932] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.133136 5215 ts_tablet_manager.cc:1888] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.33
I20240503 08:12:59.133232 5215 log.cc:1196] T b2b69fcfaa7742919bc05385b49623dd P 3482c85dd194402bbc0038963ed16d35: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-1-root/wals/b2b69fcfaa7742919bc05385b49623dd
I20240503 08:12:59.135852 4170 catalog_manager.cc:4830] TS 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611): tablet b2b69fcfaa7742919bc05385b49623dd (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:59.136834 4826 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834 [term 4 LEARNER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:12:59.137104 4432 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:12:59.137265 4355 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:12:59.137706 5185 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:59.138136 5219 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:59.139499 5185 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:59.142292 4355 consensus_queue.cc:237] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:59.143918 4752 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140 [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 08:12:59.144399 5222 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:59.146757 5222 raft_consensus.cc:2932] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.149109 4430 raft_consensus.cc:1261] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 08:12:59.149572 5203 consensus_queue.cc:1035] T 1440f42da8514e618147c69fa2795415 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:59.150738 4430 raft_consensus.cc:2932] T 1440f42da8514e618147c69fa2795415 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.151494 4753 raft_consensus.cc:2932] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.155995 4170 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1440f42da8514e618147c69fa2795415 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:59.156723 5185 raft_consensus.cc:979] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35: : Instructing follower 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:59.156826 5185 raft_consensus.cc:1067] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 4 LEADER]: Signalling peer 3482c85dd194402bbc0038963ed16d35 to start an election
I20240503 08:12:59.156638 4181 catalog_manager.cc:5484] T 1440f42da8514e618147c69fa2795415 P 9cb2634fe3314e619359e41fbb351140 reported cstate change: config changed from index 30 to 31, VOTER 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) evicted. New cstate: current_term: 3 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.158095 4359 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
from {username='slave'} at 127.0.0.1:50608
I20240503 08:12:59.158250 4359 raft_consensus.cc:482] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:59.158350 4359 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:12:59.159675 5185 raft_consensus.cc:2932] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index 30 to 31, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.160979 4359 raft_consensus.cc:504] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:59.161484 4273 tablet_service.cc:1514] Processing DeleteTablet for tablet 1440f42da8514e618147c69fa2795415 with delete_type TABLET_DATA_TOMBSTONED (TS 8df53dcdf65b407480e6d32372b48a35 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:57198
I20240503 08:12:59.162340 4359 leader_election.cc:288] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 5 election: Requested vote from peers 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947), 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:12:59.162432 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
I20240503 08:12:59.162593 4283 raft_consensus.cc:3032] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:12:59.162650 4283 raft_consensus.cc:729] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:12:59.162297 4432 raft_consensus.cc:2932] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.162683 4430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae715a227487424dbe08825a7f7d9819" candidate_uuid: "3482c85dd194402bbc0038963ed16d35" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
I20240503 08:12:59.162822 4430 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:12:59.162859 4283 consensus_queue.cc:260] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:59.163152 4283 raft_consensus.cc:3037] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:12:59.165768 4283 raft_consensus.cc:2454] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 5.
I20240503 08:12:59.166095 4325 leader_election.cc:302] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:12:59.166332 5203 raft_consensus.cc:2781] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:12:59.166435 5203 raft_consensus.cc:686] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 5 LEADER]: Becoming Leader. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:12:59.166651 5203 consensus_queue.cc:237] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } }
I20240503 08:12:59.166922 4826 raft_consensus.cc:2932] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.171095 4182 catalog_manager.cc:5484] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834 reported cstate change: config changed from index 30 to 31, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.175487 4179 catalog_manager.cc:5484] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: term changed from 4 to 5, leader changed from 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1) to 3482c85dd194402bbc0038963ed16d35 (127.0.230.2). New cstate: current_term: 5 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:59.186223 4361 raft_consensus.cc:2932] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.188505 4281 consensus_queue.cc:237] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } }
I20240503 08:12:59.190163 4826 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834 [term 4 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 08:12:59.190212 4355 raft_consensus.cc:1261] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 08:12:59.191507 5218 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:12:59.192047 5185 consensus_queue.cc:1035] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:12:59.194018 5224 raft_consensus.cc:2932] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.194305 4286 raft_consensus.cc:2932] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.196713 4430 raft_consensus.cc:2454] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482c85dd194402bbc0038963ed16d35 in term 5.
I20240503 08:12:59.197309 4169 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8ed061642594372a70df1d766d7b646 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:59.197616 4361 raft_consensus.cc:2932] T f8ed061642594372a70df1d766d7b646 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.197786 4826 raft_consensus.cc:2932] T f8ed061642594372a70df1d766d7b646 P 981e135f98fb47a1afc7fda11da0e834 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } }
I20240503 08:12:59.200160 4184 catalog_manager.cc:5484] T f8ed061642594372a70df1d766d7b646 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: config changed from index 31 to 32, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New cstate: current_term: 4 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:59.201340 5202 tablet_replica.cc:330] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35: stopping tablet replica
I20240503 08:12:59.201529 5202 raft_consensus.cc:2229] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:59.201663 5202 raft_consensus.cc:2258] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:59.202883 5202 ts_tablet_manager.cc:1875] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:59.204185 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ed061642594372a70df1d766d7b646 with delete_type TABLET_DATA_TOMBSTONED (TS 3523cd896bb842d68fbb6c15cdd1c3ed not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:34878
I20240503 08:12:59.204763 5229 tablet_replica.cc:330] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed: stopping tablet replica
I20240503 08:12:59.204924 5229 raft_consensus.cc:2229] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:59.205408 5229 raft_consensus.cc:2258] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:59.206260 5229 ts_tablet_manager.cc:1875] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:59.207986 5202 ts_tablet_manager.cc:1888] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.30
I20240503 08:12:59.208132 5202 log.cc:1196] T 1440f42da8514e618147c69fa2795415 P 8df53dcdf65b407480e6d32372b48a35: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-0-root/wals/1440f42da8514e618147c69fa2795415
I20240503 08:12:59.208762 4169 catalog_manager.cc:4830] TS 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1:32947): tablet 1440f42da8514e618147c69fa2795415 (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:59.211364 5229 ts_tablet_manager.cc:1888] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.31
I20240503 08:12:59.211484 5229 log.cc:1196] T f8ed061642594372a70df1d766d7b646 P 3523cd896bb842d68fbb6c15cdd1c3ed: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-2-root/wals/f8ed061642594372a70df1d766d7b646
I20240503 08:12:59.212041 4170 catalog_manager.cc:4830] TS 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567): tablet f8ed061642594372a70df1d766d7b646 (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:59.387027 5220 raft_consensus.cc:1050] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35: attempting to promote NON_VOTER 9cb2634fe3314e619359e41fbb351140 to VOTER
I20240503 08:12:59.387799 5220 consensus_queue.cc:237] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:59.389961 4826 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 08:12:59.390024 4430 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 08:12:59.390455 5220 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:59.390702 4753 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [term 1 LEARNER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 08:12:59.390741 5222 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:59.391103 5203 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:12:59.396489 5220 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.397042 4430 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.401088 4179 catalog_manager.cc:5484] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: config changed from index 30 to 31, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.402314 4826 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.407104 4753 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.410409 4361 consensus_queue.cc:237] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:12:59.412598 4753 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240503 08:12:59.412680 4826 raft_consensus.cc:1261] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240503 08:12:59.413127 5086 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:12:59.413384 5222 consensus_queue.cc:1035] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:12:59.415450 5220 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.415904 4826 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.416178 4753 raft_consensus.cc:2932] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } }
I20240503 08:12:59.418506 4170 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c94415b8c11d4f9790344c93eff4c331 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:59.419740 4179 catalog_manager.cc:5484] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 reported cstate change: config changed from index 31 to 32, VOTER 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3) evicted. New cstate: current_term: 1 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:59.423234 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet c94415b8c11d4f9790344c93eff4c331 with delete_type TABLET_DATA_TOMBSTONED (TS 3523cd896bb842d68fbb6c15cdd1c3ed not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:34878
I20240503 08:12:59.423545 5229 tablet_replica.cc:330] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed: stopping tablet replica
I20240503 08:12:59.423720 5229 raft_consensus.cc:2229] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:59.424083 5229 raft_consensus.cc:2258] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:59.424787 5229 ts_tablet_manager.cc:1875] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:59.428970 5229 ts_tablet_manager.cc:1888] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.31
I20240503 08:12:59.429052 5229 log.cc:1196] T c94415b8c11d4f9790344c93eff4c331 P 3523cd896bb842d68fbb6c15cdd1c3ed: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-2-root/wals/c94415b8c11d4f9790344c93eff4c331
I20240503 08:12:59.429608 4170 catalog_manager.cc:4830] TS 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567): tablet c94415b8c11d4f9790344c93eff4c331 (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:59.491828 5185 raft_consensus.cc:1050] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35: attempting to promote NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 to VOTER
I20240503 08:12:59.492725 5185 consensus_queue.cc:237] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 26, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:59.495347 4678 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 LEARNER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 08:12:59.495827 4355 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 08:12:59.495376 4753 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 08:12:59.495952 5185 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:12:59.496230 5218 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:12:59.496505 5219 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:12:59.500478 5218 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:59.501540 4753 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:59.504019 4678 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:59.503952 4355 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:59.505852 4182 catalog_manager.cc:5484] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 reported cstate change: config changed from index 31 to 32, 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:59.511026 4286 consensus_queue.cc:237] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:12:59.512650 4753 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 08:12:59.512735 4678 raft_consensus.cc:1261] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 FOLLOWER]: Refusing update from remote peer 8df53dcdf65b407480e6d32372b48a35: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 08:12:59.513192 5185 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:12:59.513494 5224 consensus_queue.cc:1035] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:12:59.515358 5185 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:59.515971 4678 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:59.516419 4753 raft_consensus.cc:2932] T d7037feb51534b57b0ac70344cdc1e18 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } }
I20240503 08:12:59.518851 4169 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d7037feb51534b57b0ac70344cdc1e18 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:59.520063 4184 catalog_manager.cc:5484] T d7037feb51534b57b0ac70344cdc1e18 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: config changed from index 32 to 33, VOTER 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) evicted. New cstate: current_term: 2 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:59.523532 4351 tablet_service.cc:1514] Processing DeleteTablet for tablet d7037feb51534b57b0ac70344cdc1e18 with delete_type TABLET_DATA_TOMBSTONED (TS 3482c85dd194402bbc0038963ed16d35 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:50578
I20240503 08:12:59.523837 5215 tablet_replica.cc:330] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35: stopping tablet replica
I20240503 08:12:59.524004 5215 raft_consensus.cc:2229] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:59.524272 5215 raft_consensus.cc:2258] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:59.524930 5215 ts_tablet_manager.cc:1875] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:59.529042 5215 ts_tablet_manager.cc:1888] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.32
I20240503 08:12:59.529150 5215 log.cc:1196] T d7037feb51534b57b0ac70344cdc1e18 P 3482c85dd194402bbc0038963ed16d35: Deleting WAL directory at /tmp/dist-test-taskOiXQOO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723866726930-920-0/minicluster-data/ts-1-root/wals/d7037feb51534b57b0ac70344cdc1e18
I20240503 08:12:59.529721 4170 catalog_manager.cc:4830] TS 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611): tablet d7037feb51534b57b0ac70344cdc1e18 (table test-workload [id=1ea0a34853d34950b2855bcc8f1879cb]) successfully deleted
I20240503 08:12:59.540117 4355 raft_consensus.cc:1261] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240503 08:12:59.540601 5192 consensus_queue.cc:1035] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:59.543789 4281 raft_consensus.cc:1261] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240503 08:12:59.544317 5192 consensus_queue.cc:1035] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:12:59.748543 4430 raft_consensus.cc:1261] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 5 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20240503 08:12:59.749027 5222 consensus_queue.cc:1035] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:12:59.751981 4281 raft_consensus.cc:1261] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 5 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20240503 08:12:59.752501 5222 consensus_queue.cc:1035] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:13:00.527673 4430 consensus_queue.cc:237] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:13:00.529549 4281 raft_consensus.cc:1261] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 08:13:00.529608 4355 raft_consensus.cc:1261] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
W20240503 08:13:00.529723 4401 consensus_peers.cc:485] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed -> Peer 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4:32895): Couldn't send request to peer 23c12ef68ccc4fad80eb918f77be34e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82e1e5d27e441ccbd62d1296c06a2f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.529982 5246 consensus_queue.cc:1035] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:13:00.530228 5192 consensus_queue.cc:1035] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:13:00.532203 5245 raft_consensus.cc:2932] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 LEADER]: Committing config change with OpId 4.30: config changed from index -1 to 30, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:13:00.532804 4355 raft_consensus.cc:2932] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index -1 to 30, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:13:00.535084 4281 raft_consensus.cc:2932] T b82e1e5d27e441ccbd62d1296c06a2f4 P 8df53dcdf65b407480e6d32372b48a35 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index -1 to 30, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:13:00.536337 4179 catalog_manager.cc:5484] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: config changed from index -1 to 30, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New cstate: current_term: 4 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:00.540293 4355 consensus_queue.cc:237] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:13:00.541998 4281 raft_consensus.cc:1261] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 5 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240503 08:13:00.542397 4430 raft_consensus.cc:1261] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 5 FOLLOWER]: Refusing update from remote peer 3482c85dd194402bbc0038963ed16d35: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240503 08:13:00.542399 5249 consensus_queue.cc:1035] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20240503 08:13:00.542657 4326 consensus_peers.cc:485] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 -> Peer 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6:39275): Couldn't send request to peer 981e135f98fb47a1afc7fda11da0e834. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae715a227487424dbe08825a7f7d9819. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.543056 5249 consensus_queue.cc:1035] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:13:00.544343 5248 raft_consensus.cc:2932] T ae715a227487424dbe08825a7f7d9819 P 3482c85dd194402bbc0038963ed16d35 [term 5 LEADER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:13:00.545533 4430 raft_consensus.cc:2932] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:13:00.547602 4281 raft_consensus.cc:2932] T ae715a227487424dbe08825a7f7d9819 P 8df53dcdf65b407480e6d32372b48a35 [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:13:00.549417 4181 catalog_manager.cc:5484] T ae715a227487424dbe08825a7f7d9819 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: config changed from index -1 to 31, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New cstate: current_term: 5 leader_uuid: "3482c85dd194402bbc0038963ed16d35" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:13:00.559944 4826 consensus_queue.cc:237] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } }
I20240503 08:13:00.562070 4355 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Refusing update from remote peer 981e135f98fb47a1afc7fda11da0e834: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:13:00.562077 4430 raft_consensus.cc:1261] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Refusing update from remote peer 981e135f98fb47a1afc7fda11da0e834: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
W20240503 08:13:00.562076 4792 consensus_peers.cc:485] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 -> Peer 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4:32895): Couldn't send request to peer 23c12ef68ccc4fad80eb918f77be34e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd99f1526db74ef5ace0a312d7afe495. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.562536 5084 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:13:00.562798 5257 consensus_queue.cc:1035] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:13:00.565011 5257 raft_consensus.cc:2932] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:13:00.565543 4430 raft_consensus.cc:2932] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:13:00.570261 4179 catalog_manager.cc:5484] T bd99f1526db74ef5ace0a312d7afe495 P 3523cd896bb842d68fbb6c15cdd1c3ed reported cstate change: config changed from index 29 to 31, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New cstate: current_term: 2 leader_uuid: "981e135f98fb47a1afc7fda11da0e834" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:13:00.571110 4430 consensus_queue.cc:237] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } }
I20240503 08:13:00.579288 4753 raft_consensus.cc:1261] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:13:00.579299 4281 raft_consensus.cc:1261] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:13:00.580099 5246 consensus_queue.cc:1035] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:13:00.580408 5192 consensus_queue.cc:1035] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:13:00.584952 4361 raft_consensus.cc:2932] T bd99f1526db74ef5ace0a312d7afe495 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } attrs { replace: true } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: NON_VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: true } } }
I20240503 08:13:00.592898 5192 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:13:00.593269 4753 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:13:00.594323 4281 raft_consensus.cc:2932] T 2d0dc42483a64c90a1492a7353e2ba4f P 8df53dcdf65b407480e6d32372b48a35 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:13:00.598239 4179 catalog_manager.cc:5484] T 2d0dc42483a64c90a1492a7353e2ba4f P 9cb2634fe3314e619359e41fbb351140 reported cstate change: config changed from index 28 to 29, NON_VOTER 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6) added. New cstate: current_term: 1 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: NON_VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: true } } }
I20240503 08:13:00.600124 4430 consensus_queue.cc:237] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } }
W20240503 08:13:00.600522 4400 consensus_peers.cc:485] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed -> Peer 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6:39275): Couldn't send request to peer 981e135f98fb47a1afc7fda11da0e834. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d0dc42483a64c90a1492a7353e2ba4f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.604525 4678 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 08:13:00.604553 4355 raft_consensus.cc:1261] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Refusing update from remote peer 3523cd896bb842d68fbb6c15cdd1c3ed: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
W20240503 08:13:00.604782 4399 consensus_peers.cc:485] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed -> Peer 9cb2634fe3314e619359e41fbb351140 (127.0.230.5:40387): Couldn't send request to peer 9cb2634fe3314e619359e41fbb351140. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03e50d2db8824a11af7a5e4f9f3ed240. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.605019 5192 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:13:00.605255 5246 consensus_queue.cc:1035] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:13:00.607450 5246 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:13:00.607968 4678 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:13:00.608593 4355 raft_consensus.cc:2932] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3482c85dd194402bbc0038963ed16d35 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:13:00.611651 4181 catalog_manager.cc:5484] T 03e50d2db8824a11af7a5e4f9f3ed240 P 23c12ef68ccc4fad80eb918f77be34e2 reported cstate change: config changed from index 32 to 33, NON_VOTER 9cb2634fe3314e619359e41fbb351140 (127.0.230.5) added. New cstate: current_term: 4 leader_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } attrs { replace: true } } peers { permanent_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed" member_type: VOTER last_known_addr { host: "127.0.230.3" port: 38567 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: NON_VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: true } } }
I20240503 08:13:00.914274 5268 ts_tablet_manager.cc:902] T b82e1e5d27e441ccbd62d1296c06a2f4 P 23c12ef68ccc4fad80eb918f77be34e2: Initiating tablet copy from peer 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:13:00.915652 5268 tablet_copy_client.cc:308] T b82e1e5d27e441ccbd62d1296c06a2f4 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.3:38567
I20240503 08:13:00.916144 4446 tablet_copy_service.cc:140] P 3523cd896bb842d68fbb6c15cdd1c3ed: Received BeginTabletCopySession request for tablet b82e1e5d27e441ccbd62d1296c06a2f4 from peer 23c12ef68ccc4fad80eb918f77be34e2 ({username='slave'} at 127.0.0.1:57524)
I20240503 08:13:00.916247 4446 tablet_copy_service.cc:161] P 3523cd896bb842d68fbb6c15cdd1c3ed: Beginning new tablet copy session on tablet b82e1e5d27e441ccbd62d1296c06a2f4 from peer 23c12ef68ccc4fad80eb918f77be34e2 at {username='slave'} at 127.0.0.1:57524: session id = 23c12ef68ccc4fad80eb918f77be34e2-b82e1e5d27e441ccbd62d1296c06a2f4
I20240503 08:13:00.918386 4446 tablet_copy_source_session.cc:205] T b82e1e5d27e441ccbd62d1296c06a2f4 P 3523cd896bb842d68fbb6c15cdd1c3ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:00.919330 5268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82e1e5d27e441ccbd62d1296c06a2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.923571 5268 tablet_copy_client.cc:783] T b82e1e5d27e441ccbd62d1296c06a2f4 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:00.923823 5268 tablet_copy_client.cc:647] T b82e1e5d27e441ccbd62d1296c06a2f4 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:00.924943 5272 tablet_copy_client.cc:1002] T b82e1e5d27e441ccbd62d1296c06a2f4 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Injecting latency into file download: 2000
I20240503 08:13:01.009115 4238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:13:01.012421 4430 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d0dc42483a64c90a1492a7353e2ba4f"
dest_uuid: "3523cd896bb842d68fbb6c15cdd1c3ed"
mode: GRACEFUL
new_leader_uuid: "8df53dcdf65b407480e6d32372b48a35"
from {username='slave'} at 127.0.0.1:34920
I20240503 08:13:01.012615 4430 raft_consensus.cc:595] T 2d0dc42483a64c90a1492a7353e2ba4f P 3523cd896bb842d68fbb6c15cdd1c3ed [term 1 LEADER]: Received request to transfer leadership to TS 8df53dcdf65b407480e6d32372b48a35
I20240503 08:13:01.013525 4355 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59aedfe83feb48d799de8eda2afb1860"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
mode: GRACEFUL
new_leader_uuid: "8df53dcdf65b407480e6d32372b48a35"
from {username='slave'} at 127.0.0.1:50644
I20240503 08:13:01.013700 4355 raft_consensus.cc:595] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Received request to transfer leadership to TS 8df53dcdf65b407480e6d32372b48a35
I20240503 08:13:01.014506 4355 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c94415b8c11d4f9790344c93eff4c331"
dest_uuid: "3482c85dd194402bbc0038963ed16d35"
mode: GRACEFUL
new_leader_uuid: "9cb2634fe3314e619359e41fbb351140"
from {username='slave'} at 127.0.0.1:50644
I20240503 08:13:01.014670 4355 raft_consensus.cc:595] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Received request to transfer leadership to TS 9cb2634fe3314e619359e41fbb351140
I20240503 08:13:01.014983 4238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 08:13:01.015179 4238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:13:01.026746 5273 ts_tablet_manager.cc:902] T bd99f1526db74ef5ace0a312d7afe495 P 23c12ef68ccc4fad80eb918f77be34e2: Initiating tablet copy from peer 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6:39275)
I20240503 08:13:01.028360 5273 tablet_copy_client.cc:308] T bd99f1526db74ef5ace0a312d7afe495 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.6:39275
I20240503 08:13:01.028878 4837 tablet_copy_service.cc:140] P 981e135f98fb47a1afc7fda11da0e834: Received BeginTabletCopySession request for tablet bd99f1526db74ef5ace0a312d7afe495 from peer 23c12ef68ccc4fad80eb918f77be34e2 ({username='slave'} at 127.0.0.1:47942)
I20240503 08:13:01.028985 4837 tablet_copy_service.cc:161] P 981e135f98fb47a1afc7fda11da0e834: Beginning new tablet copy session on tablet bd99f1526db74ef5ace0a312d7afe495 from peer 23c12ef68ccc4fad80eb918f77be34e2 at {username='slave'} at 127.0.0.1:47942: session id = 23c12ef68ccc4fad80eb918f77be34e2-bd99f1526db74ef5ace0a312d7afe495
I20240503 08:13:01.030761 4837 tablet_copy_source_session.cc:205] T bd99f1526db74ef5ace0a312d7afe495 P 981e135f98fb47a1afc7fda11da0e834: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:01.031823 5273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd99f1526db74ef5ace0a312d7afe495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.036417 5273 tablet_copy_client.cc:783] T bd99f1526db74ef5ace0a312d7afe495 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:01.036592 5273 tablet_copy_client.cc:647] T bd99f1526db74ef5ace0a312d7afe495 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:01.037931 5277 tablet_copy_client.cc:1002] T bd99f1526db74ef5ace0a312d7afe495 P 23c12ef68ccc4fad80eb918f77be34e2: tablet copy: Injecting latency into file download: 2000
I20240503 08:13:01.082567 5278 ts_tablet_manager.cc:902] T 03e50d2db8824a11af7a5e4f9f3ed240 P 9cb2634fe3314e619359e41fbb351140: Initiating tablet copy from peer 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
I20240503 08:13:01.084147 5278 tablet_copy_client.cc:308] T 03e50d2db8824a11af7a5e4f9f3ed240 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Beginning tablet copy session from remote peer at address 127.0.230.3:38567
I20240503 08:13:01.084663 4446 tablet_copy_service.cc:140] P 3523cd896bb842d68fbb6c15cdd1c3ed: Received BeginTabletCopySession request for tablet 03e50d2db8824a11af7a5e4f9f3ed240 from peer 9cb2634fe3314e619359e41fbb351140 ({username='slave'} at 127.0.0.1:34910)
I20240503 08:13:01.084833 4446 tablet_copy_service.cc:161] P 3523cd896bb842d68fbb6c15cdd1c3ed: Beginning new tablet copy session on tablet 03e50d2db8824a11af7a5e4f9f3ed240 from peer 9cb2634fe3314e619359e41fbb351140 at {username='slave'} at 127.0.0.1:34910: session id = 9cb2634fe3314e619359e41fbb351140-03e50d2db8824a11af7a5e4f9f3ed240
I20240503 08:13:01.087203 5283 raft_consensus.cc:979] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35: : Instructing follower 9cb2634fe3314e619359e41fbb351140 to start an election
I20240503 08:13:01.087385 4446 tablet_copy_source_session.cc:205] T 03e50d2db8824a11af7a5e4f9f3ed240 P 3523cd896bb842d68fbb6c15cdd1c3ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:01.087502 5249 raft_consensus.cc:1067] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Signalling peer 9cb2634fe3314e619359e41fbb351140 to start an election
I20240503 08:13:01.088232 4753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c94415b8c11d4f9790344c93eff4c331"
dest_uuid: "9cb2634fe3314e619359e41fbb351140"
from {username='slave'} at 127.0.0.1:52116
I20240503 08:13:01.088407 4753 raft_consensus.cc:482] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:01.088483 4753 raft_consensus.cc:3037] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:01.088681 5278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e50d2db8824a11af7a5e4f9f3ed240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.091291 4753 raft_consensus.cc:504] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:13:01.092146 4753 leader_election.cc:288] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [CANDIDATE]: Term 2 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 981e135f98fb47a1afc7fda11da0e834 (127.0.230.6:39275)
I20240503 08:13:01.092829 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94415b8c11d4f9790344c93eff4c331" candidate_uuid: "9cb2634fe3314e619359e41fbb351140" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:13:01.092911 5278 tablet_copy_client.cc:783] T 03e50d2db8824a11af7a5e4f9f3ed240 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:01.093010 4355 raft_consensus.cc:3032] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:01.093066 4355 raft_consensus.cc:729] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:13:01.093104 5278 tablet_copy_client.cc:647] T 03e50d2db8824a11af7a5e4f9f3ed240 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:01.093293 4355 consensus_queue.cc:260] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
I20240503 08:13:01.093605 5285 ts_tablet_manager.cc:902] T ae715a227487424dbe08825a7f7d9819 P 981e135f98fb47a1afc7fda11da0e834: Initiating tablet copy from peer 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611)
I20240503 08:13:01.093641 4355 raft_consensus.cc:3037] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 1 FOLLOWER]: Advancing to term 2
F20240503 08:13:01.093957 5285 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-03e50d2db8824a11af7a5e4f9f3ed240.queue_time_us, tablet-download-pool-03e50d2db8824a11af7a5e4f9f3ed240.queue_time_us) (tablet-download-pool-03e50d2db8824a11af7a5e4f9f3ed240.run_cpu_time_us, tablet-download-pool-03e50d2db8824a11af7a5e4f9f3ed240.run_cpu_time_us) (tablet-download-pool-03e50d2db8824a11af7a5e4f9f3ed240.run_wall_time_us, tablet-download-pool-03e50d2db8824a11af7a5e4f9f3ed240.run_wall_time_us) (tablet-download-pool-1440f42da8514e618147c69fa2795415.queue_time_us, tablet-download-pool-1440f42da8514e618147c69fa2795415.queue_time_us) (tablet-download-pool-1440f42da8514e618147c69fa2795415.run_cpu_time_us, tablet-download-pool-1440f42da8514e618147c69fa2795415.run_cpu_time_us) (tablet-download-pool-1440f42da8514e618147c69fa2795415.run_wall_time_us, tablet-download-pool-1440f42da8514e618147c69fa2795415.run_wall_time_us) (tablet-download-pool-2d0dc42483a64c90a1492a7353e2ba4f.queue_time_us, tablet-download-pool-2d0dc42483a64c90a1492a7353e2ba4f.queue_time_us) (tablet-download-pool-2d0dc42483a64c90a1492a7353e2ba4f.run_cpu_time_us, tablet-download-pool-2d0dc42483a64c90a1492a7353e2ba4f.run_cpu_time_us) (tablet-download-pool-2d0dc42483a64c90a1492a7353e2ba4f.run_wall_time_us, tablet-download-pool-2d0dc42483a64c90a1492a7353e2ba4f.run_wall_time_us) (tablet-download-pool-374e45fb23af484aa3d9c6da1f1863d3.queue_time_us, tablet-download-pool-374e45fb23af484aa3d9c6da1f1863d3.queue_time_us) (tablet-download-pool-374e45fb23af484aa3d9c6da1f1863d3.run_cpu_time_us, tablet-download-pool-374e45fb23af484aa3d9c6da1f1863d3.run_cpu_time_us) (tablet-download-pool-374e45fb23af484aa3d9c6da1f1863d3.run_wall_time_us, tablet-download-pool-374e45fb23af484aa3d9c6da1f1863d3.run_wall_time_us) (tablet-download-pool-59aedfe83feb48d799de8eda2afb1860.queue_time_us, tablet-download-pool-59aedfe83feb48d799de8eda2afb1860.queue_time_us) (tablet-download-pool-59aedfe83feb48d799de8eda2afb1860.run_cpu_time_us, tablet-download-pool-59aedfe83feb48d799de8eda2afb1860.run_cpu_time_us) (tablet-download-pool-59aedfe83feb48d799de8eda2afb1860.run_wall_time_us, tablet-download-pool-59aedfe83feb48d799de8eda2afb1860.run_wall_time_us) (tablet-download-pool-ae715a227487424dbe08825a7f7d9819.queue_time_us, tablet-download-pool-ae715a227487424dbe08825a7f7d9819.queue_time_us) (tablet-download-pool-b2b69fcfaa7742919bc05385b49623dd.queue_time_us, tablet-download-pool-b2b69fcfaa7742919bc05385b49623dd.queue_time_us) (tablet-download-pool-b2b69fcfaa7742919bc05385b49623dd.run_cpu_time_us, tablet-download-pool-b2b69fcfaa7742919bc05385b49623dd.run_cpu_time_us) (tablet-download-pool-b2b69fcfaa7742919bc05385b49623dd.run_wall_time_us, tablet-download-pool-b2b69fcfaa7742919bc05385b49623dd.run_wall_time_us) (tablet-download-pool-b82e1e5d27e441ccbd62d1296c06a2f4.queue_time_us, tablet-download-pool-b82e1e5d27e441ccbd62d1296c06a2f4.queue_time_us) (tablet-download-pool-b82e1e5d27e441ccbd62d1296c06a2f4.run_cpu_time_us, tablet-download-pool-b82e1e5d27e441ccbd62d1296c06a2f4.run_cpu_time_us) (tablet-download-pool-b82e1e5d27e441ccbd62d1296c06a2f4.run_wall_time_us, tablet-download-pool-b82e1e5d27e441ccbd62d1296c06a2f4.run_wall_time_us) (tablet-download-pool-bd99f1526db74ef5ace0a312d7afe495.queue_time_us, tablet-download-pool-bd99f1526db74ef5ace0a312d7afe495.queue_time_us) (tablet-download-pool-bd99f1526db74ef5ace0a312d7afe495.run_cpu_time_us, tablet-download-pool-bd99f1526db74ef5ace0a312d7afe495.run_cpu_time_us) (tablet-download-pool-bd99f1526db74ef5ace0a312d7afe495.run_wall_time_us, tablet-download-pool-bd99f1526db74ef5ace0a312d7afe495.run_wall_time_us) (tablet-download-pool-c94415b8c11d4f9790344c93eff4c331.queue_time_us, tablet-download-pool-c94415b8c11d4f9790344c93eff4c331.queue_time_us) (tablet-download-pool-c94415b8c11d4f9790344c93eff4c331.run_cpu_time_us, tablet-download-pool-c94415b8c11d4f9790344c93eff4c331.run_cpu_time_us) (tablet-download-pool-c94415b8c11d4f9790344c93eff4c331.run_wall_time_us, tablet-download-pool-c94415b8c11d4f9790344c93eff4c331.run_wall_time_us) (tablet-download-pool-d7037feb51534b57b0ac70344cdc1e18.queue_time_us, tablet-download-pool-d7037feb51534b57b0ac70344cdc1e18.queue_time_us) (tablet-download-pool-d7037feb51534b57b0ac70344cdc1e18.run_cpu_time_us, tablet-download-pool-d7037feb51534b57b0ac70344cdc1e18.run_cpu_time_us) (tablet-download-pool-d7037feb51534b57b0ac70344cdc1e18.run_wall_time_us, tablet-download-pool-d7037feb51534b57b0ac70344cdc1e18.run_wall_time_us) (tablet-download-pool-e85a6284cfa642e3b34d923737fb6e49.queue_time_us, tablet-download-pool-e85a6284cfa642e3b34d923737fb6e49.queue_time_us) (tablet-download-pool-e85a6284cfa642e3b34d923737fb6e49.run_cpu_time_us, tablet-download-pool-e85a6284cfa642e3b34d923737fb6e49.run_cpu_time_us) (tablet-download-pool-e85a6284cfa642e3b34d923737fb6e49.run_wall_time_us, tablet-download-pool-e85a6284cfa642e3b34d923737fb6e49.run_wall_time_us) (tablet-download-pool-f8ed061642594372a70df1d766d7b646.queue_time_us, tablet-download-pool-f8ed061642594372a70df1d766d7b646.queue_time_us) (tablet-download-pool-f8ed061642594372a70df1d766d7b646.run_cpu_time_us, tablet-download-pool-f8ed061642594372a70df1d766d7b646.run_cpu_time_us) (tablet-download-pool-f8ed061642594372a70df1d766d7b646.run_wall_time_us, tablet-download-pool-f8ed061642594372a70df1d766d7b646.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714723981 (unix time) try "date -d @1714723981" if you are using GNU date ***
I20240503 08:13:01.096276 4355 raft_consensus.cc:2454] T c94415b8c11d4f9790344c93eff4c331 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb2634fe3314e619359e41fbb351140 in term 2.
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000398) received by PID 920 (TID 0x7f9f406e6700) from PID 920; stack trace: ***
@ 0x7f9ff6bb9980 (unknown) at ??:0
I20240503 08:13:01.108075 4717 leader_election.cc:302] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482c85dd194402bbc0038963ed16d35, 9cb2634fe3314e619359e41fbb351140; no voters:
I20240503 08:13:01.108346 5038 raft_consensus.cc:2781] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:01.108942 5038 raft_consensus.cc:686] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [term 2 LEADER]: Becoming Leader. State: Replica: 9cb2634fe3314e619359e41fbb351140, State: Running, Role: LEADER
I20240503 08:13:01.109103 5282 tablet_copy_client.cc:1002] T 03e50d2db8824a11af7a5e4f9f3ed240 P 9cb2634fe3314e619359e41fbb351140: tablet copy: Injecting latency into file download: 2000
I20240503 08:13:01.109179 5038 consensus_queue.cc:237] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } }
@ 0x7f9febf06fb7 gsignal at ??:0
@ 0x7f9febf08921 abort at ??:0
I20240503 08:13:01.115100 4827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94415b8c11d4f9790344c93eff4c331" candidate_uuid: "9cb2634fe3314e619359e41fbb351140" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: true dest_uuid: "981e135f98fb47a1afc7fda11da0e834"
I20240503 08:13:01.115317 4827 raft_consensus.cc:3037] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:01.118230 4827 raft_consensus.cc:2454] T c94415b8c11d4f9790344c93eff4c331 P 981e135f98fb47a1afc7fda11da0e834 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb2634fe3314e619359e41fbb351140 in term 2.
@ 0x7f9fee9bbdcd google::LogMessage::Fail() at ??:0
I20240503 08:13:01.120191 4179 catalog_manager.cc:5484] T c94415b8c11d4f9790344c93eff4c331 P 9cb2634fe3314e619359e41fbb351140 reported cstate change: term changed from 1 to 2, leader changed from 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) to 9cb2634fe3314e619359e41fbb351140 (127.0.230.5). New cstate: current_term: 2 leader_uuid: "9cb2634fe3314e619359e41fbb351140" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "981e135f98fb47a1afc7fda11da0e834" member_type: VOTER last_known_addr { host: "127.0.230.6" port: 39275 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cb2634fe3314e619359e41fbb351140" member_type: VOTER last_known_addr { host: "127.0.230.5" port: 40387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
@ 0x7f9fee9bfb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9fee9bb7cc google::LogMessage::Flush() at ??:0
I20240503 08:13:01.128293 5294 ts_tablet_manager.cc:902] T 2d0dc42483a64c90a1492a7353e2ba4f P 981e135f98fb47a1afc7fda11da0e834: Initiating tablet copy from peer 3523cd896bb842d68fbb6c15cdd1c3ed (127.0.230.3:38567)
@ 0x7f9fee9bcf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9ff0038742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f9ff0004f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f9fefffecd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f9fff4cb8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 08:13:01.153322 5284 raft_consensus.cc:979] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35: : Instructing follower 8df53dcdf65b407480e6d32372b48a35 to start an election
I20240503 08:13:01.153474 5284 raft_consensus.cc:1067] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Signalling peer 8df53dcdf65b407480e6d32372b48a35 to start an election
I20240503 08:13:01.154072 4281 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "59aedfe83feb48d799de8eda2afb1860"
dest_uuid: "8df53dcdf65b407480e6d32372b48a35"
from {username='slave'} at 127.0.0.1:57212
I20240503 08:13:01.154249 4281 raft_consensus.cc:482] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:01.154330 4281 raft_consensus.cc:3037] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 2 FOLLOWER]: Advancing to term 3
@ 0x7f9fff4e3daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 08:13:01.157024 4281 raft_consensus.cc:504] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:13:01.157526 4281 leader_election.cc:288] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 3 election: Requested vote from peers 3482c85dd194402bbc0038963ed16d35 (127.0.230.2:44611), 23c12ef68ccc4fad80eb918f77be34e2 (127.0.230.4:32895)
I20240503 08:13:01.157866 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59aedfe83feb48d799de8eda2afb1860" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "3482c85dd194402bbc0038963ed16d35"
I20240503 08:13:01.158052 4355 raft_consensus.cc:3032] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:01.158114 4355 raft_consensus.cc:729] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3482c85dd194402bbc0038963ed16d35, State: Running, Role: LEADER
I20240503 08:13:01.158074 4678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59aedfe83feb48d799de8eda2afb1860" candidate_uuid: "8df53dcdf65b407480e6d32372b48a35" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "23c12ef68ccc4fad80eb918f77be34e2"
I20240503 08:13:01.158221 4678 raft_consensus.cc:3037] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:01.158349 4355 consensus_queue.cc:260] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
I20240503 08:13:01.158691 4355 raft_consensus.cc:3037] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 2 FOLLOWER]: Advancing to term 3
@ 0x7f9fff60e480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:13:01.160720 4678 raft_consensus.cc:2454] T 59aedfe83feb48d799de8eda2afb1860 P 23c12ef68ccc4fad80eb918f77be34e2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 3.
I20240503 08:13:01.161042 4252 leader_election.cc:302] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c12ef68ccc4fad80eb918f77be34e2, 8df53dcdf65b407480e6d32372b48a35; no voters:
I20240503 08:13:01.161273 5185 raft_consensus.cc:2781] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:01.161307 4355 raft_consensus.cc:2454] T 59aedfe83feb48d799de8eda2afb1860 P 3482c85dd194402bbc0038963ed16d35 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df53dcdf65b407480e6d32372b48a35 in term 3.
I20240503 08:13:01.161594 5185 raft_consensus.cc:686] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [term 3 LEADER]: Becoming Leader. State: Replica: 8df53dcdf65b407480e6d32372b48a35, State: Running, Role: LEADER
I20240503 08:13:01.161813 5185 consensus_queue.cc:237] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } }
@ 0x7f9fff646a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 08:13:01.165886 4179 catalog_manager.cc:5484] T 59aedfe83feb48d799de8eda2afb1860 P 8df53dcdf65b407480e6d32372b48a35 reported cstate change: term changed from 2 to 3, leader changed from 3482c85dd194402bbc0038963ed16d35 (127.0.230.2) to 8df53dcdf65b407480e6d32372b48a35 (127.0.230.1). New cstate: current_term: 3 leader_uuid: "8df53dcdf65b407480e6d32372b48a35" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "8df53dcdf65b407480e6d32372b48a35" member_type: VOTER last_known_addr { host: "127.0.230.1" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482c85dd194402bbc0038963ed16d35" member_type: VOTER last_known_addr { host: "127.0.230.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23c12ef68ccc4fad80eb918f77be34e2" member_type: VOTER last_known_addr { host: "127.0.230.4" port: 32895 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
@ 0x7f9fff629981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f9fff62970e std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa0044cb98d std::function<>::operator()() at ??:0
@ 0x7f9ff0008b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9ff000c19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9ff000c019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa0044cb98d std::function<>::operator()() at ??:0
@ 0x7f9feffe701e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9ff6bae6db start_thread at ??:0
@ 0x7f9febfe971f clone at ??:0